short course & workshop
This two-day POD short course is based on the new (2009) MIL-HDBK-1823A
"Nondestructive Evaluation System Reliability Assessment" and uses the all
new mh1823 POD software.
The training provides the latest methods for
measuring your NDE system's effectiveness as measured by Probability of
Detection (POD), and the course will use these state-of-the-art techniques
to analyze enterprise data.
Course layout is reverse-chronological – we discuss the analysis before we
discuss how to design the experiment to produce the results we are
N E W: 2017 Workshop Addendum
2017 Workshop Addendum software has been significanly expanded to provide
analysis capability for situations where min(POD) > 0 (POD "floor") and max(POD) < 1 (POD
"ceiling"). See POD "floor" and POD "ceiling"
Also new to the Addendum is the Akaike Information Criterion (AIC) and the
Schwarz Information Criterion (also known as the
Bayes Information Criterion, BIC) are used to assess
the efficacy of the added POD model floor/ceiling
In addition to supporting the new analysis
capabilities, the Workshop software allows for real-time demonstrations of
the mechanics of constructing confidence bounds on hit/miss POD vs size
We will work through examples using real data, and time will be
allocated for analyzing your enterprise-specific problems.
The usual class size is 20 participants (35 max) and, in addition to the
classroom presentations, includes the following items for each participant:
- A CD containing the R statistics computing environment and the
software and the new 2017 Workshop Addendum software, so that each participant with a Windows laptop can perform the
- Bound hard-copy of the new (2009) MIL-HDBK-1823A handbook.
- Bound hard-copy of presentation slides.
Course Content Details – Day One:
40 Years of Quantitative POD History (Understanding how we got here.)
- 1970s Have Cracks – Will Travel
- Early 1980s – Flight propulsion manufacturers’ individual efforts to improve POD analysis
- Late 1980s – USAF, UDRI, GEAE, P&W, and Allied-Signal (now Honeywell) working group produced
MIL-HDBK-1823, “Nondestructive Evaluation System Reliability Assessment” draft.
I was the editor and lead author.
- 1993 – NATO AGARD sponsored 2-day POD Short Course based on MIL-HDBK-1823 that I
presented in Ankara, Turkey, Lisbon, Portugal, Patras, Greece, and Ottawa, Canada.
- Late 1990s – USAF officially publishes MIL-HDBK-1823, 30 April, 1999
- Early 2000s – Model-Assisted POD gains a following
- February, 2007 – Draft of revised and updated MIL-HDBK-1823 released for comment,
with all-new software incorporating the latest statistical best practices for NDE data.
- 7 April, 2009 – The 2007 update was released by the USAF as
Probability and Confidence
- What is Probability? (Two incompatible definitions; both are correct)
- What is Probability of Detection?
- What is Confidence and how is that distinct from Probability?
- What is likelihood? How is it related to, but distinct from, probability?
- What does “90/95” really mean?
- Are all methods for assessing a90/95 equally effective? (Answer: No.)
- 2 kinds of NDE data. (There are more, but this is a two-day course)
How to install the mh1823 POD software and Workshop
This short-course comes with a self-contained CD with R installed along with the necessary ancillary
routines, the installed mh1823 POD software, and the example datasets – everything.
You put the CD in the drive, make a desktop icon and you’re up and running in 30 seconds.
If you already made the icon, put the CD in the drive and click the icon and you're running in
5 seconds. (We will, for completeness, spend some class time to demonstrate how to install R
from the internet, and then how to install the mh1823 POD package.)
How to analyze ‚ vs. a data
- The “ideal” POD(a) a curve
- Why ‚ vs. a data is different from Hit/Miss data
- When ‚ is less informative than simple Hit/Miss
- ‚ vs. a Data Analysis
- Read ‚ vs. a data
- Preliminary Data Assessment: Plot the data and choose the best ‚ vs. a model.
- Build the ‚ vs. a linear model
- Four ‚ vs. a Requirements (Warning: If any of these assumptions is false, or, if the model is a
line and the data describe a curve, then the subsequent POD
analysis will be wrong even though the computational steps are correct.)
- How to go from ‚ vs. a to POD vs. a – The Delta Method
- Compute the transition matrix from ‚ vs. a to POD vs. a
- The POD(a) Curve
- Wald method to compute ‚ vs. a confidence bounds
- Plot POD(a); compute POD confidence bounds
- Analyze the noise; compute the false-positive rate
- Classwork –
- Analyze a simple ‚ vs. a example.
- Effects of analysis decisions on a90/95
How to Analyze ‚ vs. a data with Repeated Measures
(Multiple inspections of the same Target Set)
- Why repeated measures are not simply “more data”
- Red apples and green apples
How to Analyze Noise
- Special Situations
- How to recognize pathological ‚ vs. a data (which is unfortunately common)
- Special difficulties with Field-Finds – When mh1823 methods are not enough
- Understanding Noise
- Definition of Noise
- Choosing a probability density to describe the noise
- False Positive Analysis (with ‚ vs. a data)
- Noise analysis and the Combined ‚ vs. a Plot
- The POD(a) Curve
- Miscellaneous mh1823 POD algorithms
Analysis of Enterprise ‚ vs. a Data
- Hands-on individual POD problem-solving
How to analyze Binary (Hit/Miss) Data
- Understanding binary data – why ordinary regression methods fail
- Read Hit/Miss data
- Build the GLM (Generalized Linear Model)
- Understanding Generalized Linear Models
- Choosing Link Functions
- Hit/Miss Confidence Bounds
- Not all statistical confidence methods are equally accurate
- How the LogLikelihood Ratio Criterion Works
- How to compute likelihood ratio confidence bounds
- Constructing Hit/Miss Confidence Bounds
- Classwork –
- Analyze a simple Hit/Miss example.
- Effects of Hit/Miss analysis decisions on
- Special Situations
- Choosing an Asymmetric Link Function
- How to analyze Repeated Measures
- How to analyze Disparate Data correctly
- How to analyze Hit/Miss Noise
- How to recognize Hit/Miss pathological data
N E W: Not covered in MIL-HDBK-1823A
– How to analyze Binary POD Floor/Ceiling Data
- How to plot max(loglikelihood ratio) as a function of a 3rd POD model parameter
- How to construct confidence bounds on the Floor or Ceiling parameter
- How to compute the Akaike Information Criterion (AIC) and the Swartz
Criterion (Bayes Information Criterion, BIC)
- How to create a real-time animated construction of confidence bounds on POD vs size curves.
Analysis of Enterprise Hit/Miss Data
Statistical Design Of eXperiments (DOX)
- What is Statistical Experimental Design?
- Variable types
- Nuisance variables
- Objective of Experimental Design
- Factorial experiments
- Categorical variables
- Noise – Probability of False Positive (PFP)
- How to Design an NDE Experiment
- Philosophy of NDE demonstrations
- How many specimens are enough?
- Specimen Design, Fabrication, Documentation, and Maintenance
- Examples of NDE Specimens
Miscellany – (Other things you should know)
- How to avoid common POD analysis Mistakes
- Model-Assisted POD (MAPOD)
- False Positives
- Sensitivity and Specificity
- Receiver Operating Characteristic (ROC) Curve
Training Review & Course Wrap-up