mh1823 POD
short course & workshop
This twoday POD short course is based on the new (2009) MILHDBK1823A
"Nondestructive Evaluation System Reliability Assessment" and uses the all
new mh1823 POD software.
The training provides the latest methods for
measuring your NDE system's effectiveness as measured by Probability of
Detection (POD), and the course will use these stateoftheart techniques
to analyze enterprise data.
Course layout is reversechronological – we discuss the analysis before we
discuss how to design the experiment to produce the results we are
analyzing.
N E W: 2017 Workshop Addendum
2017 Workshop Addendum software has been significanly expanded to provide
analysis capability for situations where min(POD) > 0 (POD "floor") and max(POD) < 1 (POD
"ceiling"). See POD "floor" and POD "ceiling"

Also new to the Addendum is the Akaike Information Criterion (AIC) and the
Schwarz Information Criterion (also known as the
Bayes Information Criterion, BIC) are used to assess
the efficacy of the added POD model floor/ceiling
parameter(s).

In addition to supporting the new analysis
capabilities, the Workshop software allows for realtime demonstrations of
the mechanics of constructing confidence bounds on hit/miss POD vs size
curves.
We will work through examples using real data, and time will be
allocated for analyzing your enterprisespecific problems.
The usual class size is 20 participants (35 max) and, in addition to the
classroom presentations, includes the following items for each participant:
 A CD containing the R statistics computing environment and the
mh1823 POD
software and the new 2017 Workshop Addendum software, so that each participant with a Windows laptop can perform the
analyses immediately.
 Bound hardcopy of the new (2009) MILHDBK1823A handbook.
 Bound hardcopy of presentation slides.
Course Content Details – Day One:
40 Years of Quantitative POD History (Understanding how we got here.)
 1970s Have Cracks – Will Travel
 Early 1980s – Flight propulsion manufacturers’ individual efforts to improve POD analysis
 Late 1980s – USAF, UDRI, GEAE, P&W, and AlliedSignal (now Honeywell) working group produced
MILHDBK1823, “Nondestructive Evaluation System Reliability Assessment” draft.
I was the editor and lead author.
 1993 – NATO AGARD sponsored 2day POD Short Course based on MILHDBK1823 that I
presented in Ankara, Turkey, Lisbon, Portugal, Patras, Greece, and Ottawa, Canada.
 Late 1990s – USAF officially publishes MILHDBK1823, 30 April, 1999
 Early 2000s – ModelAssisted POD gains a following
 February, 2007 – Draft of revised and updated MILHDBK1823 released for comment,
with allnew software incorporating the latest statistical best practices for NDE data.
 7 April, 2009 – The 2007 update was released by the USAF as
MILHDBK1823A.
Probability and Confidence
 What is Probability? (Two incompatible definitions; both are correct)
 What is Probability of Detection?
 What is Confidence and how is that distinct from Probability?
 What is likelihood? How is it related to, but distinct from, probability?
 What does “90/95” really mean?
 Are all methods for assessing a_{90/95} equally effective? (Answer: No.)
 2 kinds of NDE data. (There are more, but this is a twoday course)
How to install the mh1823 POD software and Workshop
Addendum software
This shortcourse comes with a selfcontained CD with R installed along with the necessary ancillary
R
routines, the installed mh1823 POD software, and the example datasets – everything.
You put the CD in the drive, make a desktop icon and you’re up and running in 30 seconds.
If you already made the icon, put the CD in the drive and click the icon and you're running in
5 seconds. (We will, for completeness, spend some class time to demonstrate how to install R
from the internet, and then how to install the mh1823 POD package.)
How to analyze â vs. a data
 Background
 The “ideal” POD(a) a curve
 Why â vs. a data is different from Hit/Miss data
 When â is less informative than simple Hit/Miss
 â vs. a Data Analysis
 Read â vs. a data
 Preliminary Data Assessment: Plot the data and choose the best â vs. a model.
 Build the â vs. a linear model
 Four â vs. a Requirements (Warning: If any of these assumptions is false, or, if the model is a
line and the data describe a curve, then the subsequent POD
analysis will be wrong even though the computational steps are correct.)
 How to go from â vs. a to POD vs. a – The Delta Method
 Compute the transition matrix from â vs. a to POD vs. a
 The POD(a) Curve
 Wald method to compute â vs. a confidence bounds
 Plot POD(a); compute POD confidence bounds
 Analyze the noise; compute the falsepositive rate
 Classwork –
 Analyze a simple â vs. a example.
 Effects of analysis decisions on a_{90/95}
How to Analyze â vs. a data with Repeated Measures
(Multiple inspections of the same Target Set)
 Why repeated measures are not simply “more data”
 Red apples and green apples
 Special Situations
 How to recognize pathological â vs. a data (which is unfortunately common)
 Special difficulties with FieldFinds – When mh1823 methods are not enough
How to Analyze Noise
 Understanding Noise
 Definition of Noise
 Choosing a probability density to describe the noise
 False Positive Analysis (with â vs. a data)
 Noise analysis and the Combined â vs. a Plot
 The POD(a) Curve
 Miscellaneous mh1823 POD algorithms
Analysis of Enterprise â vs. a Data
 Handson individual POD problemsolving
Day Two:
How to analyze Binary (Hit/Miss) Data
 Understanding binary data – why ordinary regression methods fail
 Read Hit/Miss data
 Build the GLM (Generalized Linear Model)
 Understanding Generalized Linear Models
 Choosing Link Functions
 Hit/Miss Confidence Bounds
 Not all statistical confidence methods are equally accurate
 How the LogLikelihood Ratio Criterion Works
 How to compute likelihood ratio confidence bounds
 Constructing Hit/Miss Confidence Bounds
 Classwork –
 Analyze a simple Hit/Miss example.
 Effects of Hit/Miss analysis decisions on
a_{90/95}
 Special Situations
 Choosing an Asymmetric Link Function
 How to analyze Repeated Measures
 How to analyze Disparate Data correctly
 How to analyze Hit/Miss Noise
 How to recognize Hit/Miss pathological data
N E W: Not covered in MILHDBK1823A
– How to analyze Binary POD Floor/Ceiling Data
 How to plot max(loglikelihood ratio) as a function of a 3rd POD model parameter
 How to construct confidence bounds on the Floor or Ceiling parameter
 How to compute the Akaike Information Criterion (AIC) and the Swartz
Criterion (Bayes Information Criterion, BIC)
 How to create a realtime animated construction of confidence bounds on POD vs size curves.
Analysis of Enterprise Hit/Miss Data
Statistical Design Of eXperiments (DOX)
 What is Statistical Experimental Design?
 Variable types
 Nuisance variables
 Objective of Experimental Design
 Factorial experiments
 Categorical variables
 Noise – Probability of False Positive (PFP)
 How to Design an NDE Experiment
 Philosophy of NDE demonstrations
 How many specimens are enough?
 Specimen Design, Fabrication, Documentation, and Maintenance
 Examples of NDE Specimens
Miscellany – (Other things you should know)
 How to avoid common POD analysis Mistakes
 ModelAssisted POD (MAPOD)
 False Positives
 Sensitivity and Specificity
 Receiver Operating Characteristic (ROC) Curve
Training Review & Course Wrapup