This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS). Verification of Performance Specifications An Advanced View of Method Validation Version 5.0, August 2012
Verification of Performance Specifications. An Advanced View of Method Validation V ersion 5.0, August 2012. Objectives. Identify test classifications Define what each validation experiment details for testing methods - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS).
Verification of Performance Specifications
An Advanced View of Method Validation
Version 5.0, August 2012
2
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
Objectives
3
A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. Food and Drug Administration (FDA)-approved,
Random Error Mean Standard deviation (SD) Coefficient of variation (CV)
Systematic and Random Errors
23
Tools for Use
Spreadsheets with
calculationsValidation Software
(Westgard, Analyze-It, EP Evaluator)
Statistical calculators, graph paper
Data-Crunching
Tools
24
One quantitative test taken through the validation process
One qualitative method taken through the validation process
How We Will Work Through This Module
Reportable Range
Precision
Accuracy
Reference Intervals
Sensitivity
Specificity
Elements of Validation
25
Repeat testing over short and long term (one day and 20 days, respectively)
20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)
Standard solutions Control materials Pools (short term only)
Precision Definition: Reproducibility Gives information related to random error
Introduction
What is needed
How we perform
the testing
26
27
Precision: How We Evaluate the Data
Mean Standard deviation (SD) Coefficient of Variation (CV)
Short term: 0.25 of allowable total error Long term: 0.33 of allowable total error
Calculate the following:
What amount of random error is allowable, based on CLIA criteria?
28
Link for: Clinical Laboratory Improvement Amendments (CLIA) College of American Pathologists (CAP) Royal College of Pathologists of Australasia (RCPA) Others
http://www.dgrhoads.com/db2004/ae2004.php
Allowable Total Error Database
29
Precision: Levey-Jennings (LJ) ChartsVa
lues
Run
30
Precision: How We Evaluate the Data
Mean SD CV: More commonly used, allows for
easier comparison
How do we compare to manufacturer’s data?
31
Precision ExampleMean of Level 1 Glucose
CLIA Total Allowable Error
Total Allowable Error Level 1 Glucose
Random error allowed:
90 mg/dL
6 mg/dL or ± 10%
0.1 x 90 = 9 mg/dL
0.25 x total allowable
0.25 x 9 mg/dL
2.25 mg/dL
0.33 x total allowable
0.33 x 9 mg/dL
2.97 mg/dL
Long-term precision
Short-term precision
32
Work with Levey-Jennings graph and data Work with mean and standard deviation to calculate a
coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation
Determine if precision data is acceptable
Activity
Accuracy Definition: How close to the true value Comparison of methods Gives information related to systematic error Potential conflicts on interpretation of results
(reference values)
Introduction
40 different specimens Cover reportable range of method Quality versus quantity
What is needed
Duplicate measurements of each specimen on each method
Minimum of five days, prefer over 20 (since replicate testing is same)
How we perform the
testing
33
Accuracy: How We Evaluate the Data
Graph the Data:
Test method on Y-axisReference (comparative) method on X-axisShows analytical range of data, linearity of response over range and relationship between methods
Real time Difference plot
Comparison plot Calculate y = mx + b
b represents constant error
m represents proportional error
34
35
Visual Inspection for AccuracyTe
st M
etho
d
Reference Method
Intercept
(x1, y1)
(x2, y2)Slope = (y2- y1) / (x2- x1)
36
Slope: Usually not significantly different from 1 Intercept: Not significantly different from 0 Significant difference with Medical Decision Points
Accuracy: How We Evaluate the Data
37
Slope Measure of proportional bias
m = (y1-y2)/(x1-x2) or “rise/run” Slope greater than 1 means the Y (Test) values are
generally higher than the X (Comparative) values Slope of 1.11 means the Y (Test) values are on
average 11% higher than the X (Comparative) values
Calculate Appropriate Statistics
38
Intercept of the Line Measure of constant bias between two methods
Y (Test) value at the point where the line crosses the Y axis
If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values
Calculate Appropriate Statistics (cont'd)
39
Accuracy
What type of bias do you see?
Accuracy (cont’d)
Constant Bias Proportional Bias
40
41
Can a linear regression formula offer predictive value in relation to method comparisons?
A. YesB. No
Skill Check
42
Create graph based on sample set Determine slope from best-fit line Determine Y-intercept from best-fit line Explain the relationship between comparative and test
results
Activity
CLSI recommends four measurements of each specimen; three are sufficient
Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)
Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens
At least four levels (five preferred)
Reportable Range / Linearity Definition: Lowest and highest test results that
are reliable Especially important with two point calibrations Analytical Measurement Range (AMR) and
derived Clinical Reportable Range (CRR)
Introduction
What is needed
How we perform the
testing
43
44
Reportable Range:How We Evaluate the Data
Measured values on Y-axis versus Known or assigned values on X-axis
Plot mean values of:
Compare with expected values (typically provided by manufacturer)
Visually inspect, draw best-fit line, estimate reportable range
45
Reportable Range Activity
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 ____ 11.0 10.0 11.0 10.0
100.0 ____ 99.0 103.0 103.0 101.0
300.0 ____ 303.0 305.0 304.0 306.0
500.0 ____ 505.0 506.0 505.0 506.0
800.0 ____ 740.0 741.0 744.0 742.0
46
Reportable Range Activity (cont'd)
AssignedValue
Experimental Results
Average Rep #1 Rep #2 Rep #3 Rep #4
10.0 10.5 11.0 10.0 11.0 10.0
100.0 101.5 99.0 103.0 103.0 101.0
300.0 304.5 303.0 305.0 304.0 306.0
500.0 505.5 505.0 506.0 505.0 506.0
800.0 741.8 740.0 741.0 744.0 742.0
Reportable Range Activity (cont'd)Linearity Scatter Plot
0.0
100.0
200.0
300.0
400.0
500.0
600.0
700.0
800.0
0 100 200 300 400 500 600 700 800 900
Assigned Concentrations (units)
Rec
over
ed V
alue
s (M
eans
)
47
AMR vs. CRR
Analytical Measurement Range (AMR)
Linearity
Clinically Reportable Range (CRR)
Allows for dilution or other preparatory steps beyond routine
48
49
If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely?
A. AMRB. CRRC. Neither A or BD. Both A and B
Skill Check
50
Utilizing the marketing materials from the two chemistry linearity kits in your handouts:
1. Determine which kit would be more appropriate for use with the chemistry assay you chose earlier
Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide.
1. Review data; record any initial observations2. Graph data on supplied graph paper3. Determine your assay’s AMR
Using an Excel spreadsheet, create a graph and calculate
linear regression statistics from the data provided
53
54
Rep 1 Rep 2 Rep 3 Rep 4Lab's
AverageKnownConc
24 23 25 24 24 25
196 197 171 194 195.7 200
359 360 358 361 360 375
530 532 529 535 532 550
700 695 702 709 702 725
55
0100200300400500600700800
0 200 400 600 800
Reco
vere
d
Known Concentration
AMR Verfication
56
Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is15 – 1400 mg/dL
Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP
Calculate sensitivity and specificity and compare to manufacturer
Qualitative Assays
67
Negative and Positive Quality Controls Use QC materials recommended by manufacturer for
verification purposes Determine validity of other results, e.g., method
comparisons Evaluate failed runs if they occur during verification
process
Qualitative Assays: Control of Validation
68
How is it performed? Runs of specimens with analyte concentrations near
the cutoff point Three specimens, one at cutoff, one just below cutoff,
and one just above cutoff (± 20% recommended) Replicate measurements of each of three specimens
(20 each, minimum) How is it evaluated?
Determine percentage of positives and negatives for each specimen
Evaluate cutoff, as well as other two specimens
Qualitative Methods: Precision
69
How is it performed? Specimens typical of population (to be tested in future
use of method) 50 positive specimens and 50 negative specimens
recommended; minimum 20 each Performed over 10 to 20 days
How is it evaluated? Discrepant results near cutoff? Most often sensitivity and specificity used to describe
performance
Accuracy/Method Comparisons
70
Qualitative MethodsComparative or Reference
Method Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
False Positive Rate - False Positives divided by total number of Negatives
False Negative Rate - False Negatives divided by total number of Positives
True vs. False
71
Qualitative Methods (cont'd) Comparative or Reference
Method Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)
Specificity = 100 x True Negatives divided by (True Negatives + False Positives)
72
Qualitative Methods (cont'd)Comparative or Reference
Method Result
Positive Negative
Test Method Result
Positive True Positive False Positive Positive Predictive Value
Negative False Negative True NegativeNegative
PredictiveValue
Sensitivity Specificity
Predictive Values - Operation of a test on a mixed population of Positive and Negatives
A property of the test and the population; and affected by prevalence of Positives
Positive Predictive Value = True Positives divided by (True Positives + False
Positives) Negative Predictive Value = True Negatives divided by
(True Negatives + False Negatives)
73
High Diagnostic Value 100% Sensitivity 100% Specificity
What happens if True Positive rate is equal to the False Positive rate?
Evaluation Criteria
74
Activity
Estimate sensitivity and specificity of a qualitative method given a data set.
75
Activity (cont’d)
Create a validation plan for a quantitative assay to be performed in your laboratory.
76
Now that you have completed this module, you should be able to:
Identify test classifications Define what each validation experiment details for
testing methods Discuss what is recommended to perform each of the
validation experiments for testing methods Recognize how to evaluate data obtained from each of
the validation experiments
In Closing
77
A rapid HIV test would likely be classified as a:
A. High complexity, modified assayB. Moderate complexity, unmodified assayC. FDA-approved, modified assayD. Waived, FDA-approved, unmodified assay
Post-Assessment Question #1
78
The precision of a test method gives information related to the method’s:
A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and
icterusE. Both A and B
Post-Assessment Question #2
79
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?
A. 20B. 18C. 16D. 15
Post-Assessment Question #3
80
Which linear regression equation component gives information regarding constant bias?
A. yB. xC. m (slope)D. b (intercept)
Post-Assessment Question #4
81
DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines. www.westgard.com. Validation of Qualitative Methods. 42 CFR § 493.1253. College of American Pathologists Commission on Laboratory Accreditation,
Accreditation Checklists, April 2006. Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard
QC, Inc., 2003. Clinical and Laboratory Standards Institute. User Protocol for Evaluation of
Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002.
Clinical and Laboratory Standards Institute. Evaluation of Precision. Performance of Quantitative Measurement Methods. NCCLS document EP5-A2.
Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004. Clinical and Laboratory Standards Institute. User verification of Performance for
Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005.