CITAC 2002-1/ 1 Identification, Measurement and Decision in Analytical Chemistry Steve Ellison LGC, England.

/CITAC 2002-1/ 1

Identification, Measurement and Decision in Analytical Chemistry

Steve EllisonLGC, England

/CITAC 2002-1/ 2

Introduction

What is Identification?

Why does it matter?

Where does measurement fit in?

Quality in identification

How sure are you?– characterising uncertainty and method

performance

/CITAC 2002-1/ 3

What is identification?

“Classification according to specific criteria”– “Above” or “Below” a limit– “Within Spec.”– “Red”– Classification into ranges (<2; 2-5; 5-10; >10)– Molecular species by NMR, IR, MS…..– Material or ingredient (“Rubber”, “Fat”…)– Origin or authenticity

/CITAC 2002-1/ 4

Why does it matter?

Classification contributes to decisions

Decisions cost money– Incorrect batch rejection incurs reprocessing costs– Incorrect acceptance risks litigation and loses

business– False positives may generate spurious prosecutions

Costs are directly related to false classification probabilities– Know probabilities - optimise cost

/CITAC 2002-1/ 5

Where does measurement fit in?

Measurement contributes to most identifications– Comparison with limits– Consistency of values (wavelength, mass,

sequence length…)

But not all– Relative Pattern identification (?)– Colour matching by eye– Identity parades….

/CITAC 2002-1/ 6

Interpretation Against Limits

Measurement result

(a)

(b) (c)

(d) (e)

(f) (g)

Limit

/CITAC 2002-1/ 7

Controlling Identification

Good practice guidance

Stated criteria

Trained staff

Controlled and calibrated instruments– Traceability!

Validated methods

….. etc

/CITAC 2002-1/ 8

How sure are you?

Does Measurement Uncertainty apply?

If not, what does?

/CITAC 2002-1/ 9

Does Measurement Uncertainty Apply?

NONO

at least, not for the ‘classification’ result

/CITAC 2002-1/ 10

Uncertainty and classification

When is the limit exceeded?

(a)

(b) (c)

(d) (e)

(f) (g)

Uncertainty in the measurement result

contributes to uncertainty about classification

/CITAC 2002-1/ 11

Uncertainty and classification

Uncertainty in the measurement result contributes to uncertainty about classification

Uncertainties in test conditions lead to uncertainty in classification

Uncertainties should be controlled to have little effect on the test result

/CITAC 2002-1/ 12

Characterising ‘uncertainty’ in identification False response rates

– What is a false response rate?– How is it determined?

Alternative expressions of method performance or uncertainty

/CITAC 2002-1/ 13

False response rates

Negative Positive

Negative TN FP

Positive FN TP

Observed

Act

ual

/CITAC 2002-1/ 15

False negative rates

Fraction of observed negatives which are false

Fraction of true positives reading negative

Fraction of all results which are incorrectly read as negative

/CITAC 2002-1/ 16

False negative rates

Fraction of observed negatives which are false

Fraction of true positives reading negative*

Fraction of all results which are incorrectly read as negative *AOAC Definition

(clinical)

/CITAC 2002-1/ 17

False negative rates

Fraction of observed negatives which are false

Fraction of true positives reading negative*

Fraction of all results which are incorrectly read as negative$

*AOAC Definition(clinical)

$The one that affects costs directly

/CITAC 2002-1/ 20

False response rates: Example*

Negative Positive

Negative 422 4

Positive 7 119

Observed

Act

ual

*EMIT test for cocaine in urine: Ferrara et al, J. Anal. Toxicol., 1994, 18, 278

/CITAC 2002-1/ 21

False negative rates: Example

Fraction of observed negatives which are false– 7/429 = 1.6%

Fraction of true positives reading negative– 7/126 = 5.6%

Fraction of all results which are incorrectly read as negative– 7/522 = 1.3%

/CITAC 2002-1/ 22

False response rates - how much data? Observed: 7/126 (5.6%)

95% confidence interval (binomial)– 1.6% to 9.5%

95% CI proportional to 1/nobs

– needs a LOT of false responses for precise figures– but false responses are rare for good methods….

Most useful direct studies are ‘worst case’ or near 50% false response levels

/CITAC 2002-1/ 23

False responses: Estimation from thresholds

Concentration

Frequency

0

x1 = 0

x2 > 0

"positive""negative"

threshold

falsepositiveregion

falsenegativeregion

/CITAC 2002-1/ 24

False responses: From probabilities Spectroscopic identification study

– S.L.R. Ellison, S.L. Gregory, Anal. Chim. Acta., 1998 370 181.

Calculated chance FT-IR match probabilities– probabilities based on “match-binning” - hits

within set distance– required hypergeometric distribution (n matches of

m taken from population)

Compared with actual hits on IR database

/CITAC 2002-1/ 25

False responses: From probabilities Theoretical predictions very sensitive to

probability assumptions – 10% changes in p make large differences in

predictions

Best performance within factor of 3-10– (Improved over binomial probabilities by >106)

Probability information must be excellent for good predictions

/CITAC 2002-1/ 26

False response rates from databases Most spectral databases contain 1 of each

material– most populations do not!

Population data must account for sub-populations– cf. DNA profiles for racially inhomogeneous

populations

/CITAC 2002-1/ 27

Alternative performance indicators

Reliability Measure Formula

False positive rate FP / (TN + FP)

False negative rate FN / (TP + FN)

Sensitivity TP / (TP + FN )

Specificity TN / (TN + FP)

Efficiency (TP + TN) / (TP + TN + FP + FN)

Youden Index Sensitivity + Specificity - 100

Likelihood ratio (1-False neg. rate)/(False pos. rate)

Bayes posterior probability Bayes rule (requires ‘prior’)- valuable for cumulative data

/CITAC 2002-1/ 28

Conclusions

Classification needs control to save money

INPUT uncertainties need control

False response rates and derived measures are useful performance indicators– Definitions vary and make a (big) difference– Sufficient data are hard to get except for carefully chosen

analyte levels– Databases suspect unless built for the purpose– Theoretical predictions usable with great care

Unwise to expect precise numbers!

/CITAC 2002-1/ 29

Best practice

Consider costs of false responses

Control qualitative test conditions via traceable calibration of equipment

Check most critical false response rate– preferably both

Use ‘worst-case’ and likely interferent studies to show limits of method performance

Use APPROPRIATE population data

Report with caution– particularly on probability estimates

/CITAC 2002-1/ 30