Comparability of Comparability of Laboratory Tests Laboratory Tests James O. Westgard James O. Westgard University of Wisconsin Medical School University of Wisconsin Medical School Westgard QC, Inc. Westgard QC, Inc. Madison, WI Madison, WI www.westgard.com www.westgard.com 1 1
51
Embed
Comparability ofLaboratory Tests - James O. Westgard
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Comparability of Comparability of Laboratory TestsLaboratory Tests
James O. WestgardJames O. WestgardUniversity of Wisconsin Medical SchoolUniversity of Wisconsin Medical School
Westgard QC, Inc.Westgard QC, Inc.Madison, WIMadison, WI
““A rose is a rose is a rose!A rose is a rose is a rose!””
22
But, is a But, is a ““glucose a glucose a glucose a glucose a glucose?glucose?””
Or, is a glucose a BGM glucose Or, is a glucose a BGM glucose or an eor an e--glucose from HbA1c?glucose from HbA1c?
Or, is a GHb a HbA1c or an Or, is a GHb a HbA1c or an
NGSP certified HbA1c?NGSP certified HbA1c?33
Harmonization is a major issue Harmonization is a major issue for medical laboratories today!for medical laboratories today!
““Improving Clinical Laboratory Testing Through Improving Clinical Laboratory Testing Through Harmonization: An International ForumHarmonization: An International Forum””, Oct 26, Oct 26--
““The AACC is hosting an The AACC is hosting an invitation onlyinvitation only conference conference on global harmonization of results from clinical on global harmonization of results from clinical laboratory testing procedures for which no reference laboratory testing procedures for which no reference measurement procedure exists or is likely to be measurement procedure exists or is likely to be developed. The twodeveloped. The two--day conference will seek to day conference will seek to strengthen quality of laboratory measurements and strengthen quality of laboratory measurements and improve patient care by developing consensus on improve patient care by developing consensus on technical and organizational processes to achieve technical and organizational processes to achieve harmonization of clinical laboratory results. harmonization of clinical laboratory results.
44
HarmonizationHarmonization
CLSI definition: CLSI definition: harmonization harmonization in glycohemoglobin (GHB) testing, the in glycohemoglobin (GHB) testing, the process by which GHB test results among process by which GHB test results among laboratories are made comparable to a laboratories are made comparable to a common reference. common reference.
Harmonization is a process!Harmonization is a process!
Comparability is a measure of the outcome!Comparability is a measure of the outcome!
Are HbA1c test results comparable today?Are HbA1c test results comparable today?
55
Need for Comparability of Need for Comparability of Medical Laboratory TestsMedical Laboratory Tests
Patients being treated in a healthcare Patients being treated in a healthcare organization that includes doctors offices & organization that includes doctors offices & clinics & hospitals, each having their own clinics & hospitals, each having their own laboratory testing servicelaboratory testing service
Patients being treated in multiple healthcare Patients being treated in multiple healthcare organizations organizations
In a geographic regionIn a geographic region
Throughout a countryThroughout a country
Throughout the worldThroughout the world66
Need for ComparabilityNeed for Comparability
Electronic Medical RecordElectronic Medical Record
Merging patient information from many Merging patient information from many different sources, including different different sources, including different laboratorieslaboratories
Assumes that Assumes that ““A glucose is a glucose is a A glucose is a glucose is a glucoseglucose””, but we know that isn, but we know that isn’’t true!t true!
Blood glucose meter results from patientBlood glucose meter results from patient
PointPoint--ofof--Care results from Doctors OfficesCare results from Doctors Offices
Hospital results from medical laboratoriesHospital results from medical laboratories
77
Need for ComparabilityNeed for Comparability
New New ““PayPay--forfor--PerformancePerformance””
incentives for incentives for quality improvementquality improvement
Chronic diseases of high priorityChronic diseases of high priority
Likely to utilize laboratory tests as measures Likely to utilize laboratory tests as measures of performanceof performance
Treatment goals may assume absolute Treatment goals may assume absolute measures that are comparable from method measures that are comparable from method to method, lab to lab, and over timeto method, lab to lab, and over time
88
How assure comparability?How assure comparability?
TraceabilityTraceability
In principle, laboratory methods should have In principle, laboratory methods should have common reference materials and methods to common reference materials and methods to establish the correct or true test valuesestablish the correct or true test values
In practice, test performance must be In practice, test performance must be monitored to demonstrate comparability monitored to demonstrate comparability
99
How assure comparability?
1010
BQC3Ch 18P 281
(3) Validate MethodPerformance (CV,bias)
(1) Define Goals for Intended Use
(TEa, Dint)
(3a) Manufacturer’sClaims
(1a) Regulatory &AccreditationRequirements
(1b) Clinical and Medical Applications
(2) Select AnalyticMeasurement
Procedure
(2a) Traceability& Calibration
(2b) Manufacturer’sReference Methods
& Materials
(4) Design SQC(rules, N, F)
(5) FormulateAQC Strategy
(11) Improve AQCEffectiveness
(5b) Lab Evaluationof Residual Risk
(5a) Manufacturer’sRisk Analysis
(6) Develop AQC Plan
(10) Monitor AQCEffectiveness (f), EQA
(7) Implement AQC System
(6a) QC Toolbox
(8) Verify Attainmentof Intended Quality
of Test Results
(9) Measure Quality& Uncertainty
Traceability Traceability ––
not a new conceptnot a new concept
Guidelines from the Guild of Soup MakersGuidelines from the Guild of Soup Makers
““Every new master sees to it that the guild weights Every new master sees to it that the guild weights and measures are renewed on his appointment. The and measures are renewed on his appointment. The old ones are then stored for comparison. old ones are then stored for comparison.
““If the measures are the same, the ingredients must If the measures are the same, the ingredients must also be the same. Our soups do not merely conform also be the same. Our soups do not merely conform with the standards, they are part of the standards.with the standards, they are part of the standards.
““An unbroken line that comes down to us from the An unbroken line that comes down to us from the days of the conquest.days of the conquest.””
Jason Goodwin. The Janissary Tree. Pickador Books, Jason Goodwin. The Janissary Tree. Pickador Books, NY, p 38. NY, p 38. A mystery story set in the 1800s in the A mystery story set in the 1800s in the Ottoman Empire featuring Investigator Yashim.Ottoman Empire featuring Investigator Yashim.
1111
Metrological Standards for Metrological Standards for Quality Measurement ProcessesQuality Measurement Processes
TraceabilityTraceability
Property of the results of a measurement or the value Property of the results of a measurement or the value of a standard whereby it can be related to stated of a standard whereby it can be related to stated references, usually national or international references, usually national or international standards, through an standards, through an unbroken chain of unbroken chain of comparisons comparisons all having stated uncertaintiesall having stated uncertainties
Uncertainty of measurementUncertainty of measurement
Parameter, associated with the result of a Parameter, associated with the result of a measurement, that characterizes the dispersion of the measurement, that characterizes the dispersion of the values that could reasonably be attributed to the values that could reasonably be attributed to the measurandmeasurand
1212
Traceability ChainTraceability Chain
1313
Issues with TraceabilityIssues with Traceability
Definition of analyte and unitsDefinition of analyte and units
Analyte may be a class of substancesAnalyte may be a class of substances
Lack of specificity in measurement proceduresLack of specificity in measurement procedures
Need for traceability models that utilize Need for traceability models that utilize ““acceptedaccepted””
reference materials and methods, reference materials and methods,
valuevalue--assignment protocols, laboratory assignment protocols, laboratory networks as base of referencenetworks as base of reference
1515
Model II: International Reference Model II: International Reference Method and CalibratorMethod and Calibrator
1616
““Medical TraceabilityMedical Traceability””
of HbA1cof HbA1cIFCC Reference
Method
Value-assignment of Calibrator Materials National Reference
Methods (US NGSP)NGSP Certification
Program ManufacturersMethods
Manufacturers Calibrators Laboratory
MethodsTest
Samples
Test Results
CAPPT
IFCC LabNetwork
1717
Comparability in the Real WorldComparability in the Real World HbA1c exampleHbA1c example
LentersLenters--Westra E, Slingerland RJ. Six of Westra E, Slingerland RJ. Six of Eight Hemoglobin A1c PointEight Hemoglobin A1c Point--ofof--Care Care Instruments Do Not Meet the General Instruments Do Not Meet the General Accepted Analytical Performance Accepted Analytical Performance Criteria. Clin Chem 2010;56:44Criteria. Clin Chem 2010;56:44--52.52.
Bruns DE, Boyd JC. Few PointBruns DE, Boyd JC. Few Point--ofof--Care Care Hemoglobin A1c Assay Methods Meet Hemoglobin A1c Assay Methods Meet Clinical Needs. Clin Chem 2010;56:4Clinical Needs. Clin Chem 2010;56:4--6.6.
1818
POC POC ““Waived TestWaived Test””
FDA review requires manufacturer to FDA review requires manufacturer to define Allowable Total Error to validate define Allowable Total Error to validate quality of quality of ““waivedwaived””
teststests
Recommends use of Recommends use of ““error gridserror grids””
Clarke et al. Diabetes Care 1987;10:622Clarke et al. Diabetes Care 1987;10:622--88
Parkes et al. Diabetes Care 2000;23:1143Parkes et al. Diabetes Care 2000;23:1143--88
CLSI EP27P published late 2009CLSI EP27P published late 2009
““How to Construct and Interpret an Error Grid How to Construct and Interpret an Error Grid for Diagnostic Assaysfor Diagnostic Assays””
1919
FDA Guidance (2008)FDA Guidance (2008) Allowable Total Error GridAllowable Total Error Grid
Guidance for Industry and FDA Staff: Recommendations for Clinical LaboratoryImprovement Amendments of 1988 (CLIA) Waiver Applications for Manufacturers’of Invitro Diagnostic Devices. Jan 30, 2008, Food and Drug Adminstration 2020
Calculation of Calculation of ““estimatedestimated--GlucoseGlucose””
ADA recommends conversion of HbA1c ADA recommends conversion of HbA1c results to results to ““estimated glucoseestimated glucose””
to help to help
patients understand the clinical implicationspatients understand the clinical implications
ISO standard for Blood Glucose MetersISO standard for Blood Glucose Meters
20% TEa or 15 mg/dL < 75 mg/dL20% TEa or 15 mg/dL < 75 mg/dL
CLIA criterion for acceptable performanceCLIA criterion for acceptable performance
6 mg/dL or 10%, whichever is larger6 mg/dL or 10%, whichever is larger
HbA1c Requirements forHbA1c Requirements for Monitoring Glycemic ControlMonitoring Glycemic Control
2121
“e-Glucose”
Requirements ISO 15197 vs CLIA
2222
Quality Quality ‘‘StandardsStandards’’
for BMG for BMG Devices may change! 15%???Devices may change! 15%???
95% agreement within specified limits95% agreement within specified limits
±±
0.85 %Hb in 20090.85 %Hb in 2009
±±
0.75 %Hb in 20100.75 %Hb in 2010
CAP proficiency testing requirementCAP proficiency testing requirement
TEa of 10% in 2009TEa of 10% in 2009
TEa of 8.0% in 2010TEa of 8.0% in 2010
TEa of 6.0% in 2011TEa of 6.0% in 2011
NGSP and CAP CriteriaNGSP and CAP Criteria
2424
2010 NGSP vs CAP Criteria 2010 NGSP vs CAP Criteria WhatWhat’’s wrong with this picture?s wrong with this picture?
2525
HbA1c Requirement for HbA1c Requirement for Diagnosis of DiabetesDiagnosis of Diabetes
ADA recommends cutoff of 6.5 %HbADA recommends cutoff of 6.5 %Hb
Values from 5.7 to 6.4 %Hb higher riskValues from 5.7 to 6.4 %Hb higher risk
Values of 5.6 %Hb and less Values of 5.6 %Hb and less ““normalnormal””
Interpretation for error gridInterpretation for error grid
False NegativesFalse Negatives: Values of 6.5 %Hb that : Values of 6.5 %Hb that are erroneously measured as 5.6 %Hb or lessare erroneously measured as 5.6 %Hb or less
False PositivesFalse Positives: Values of 5.6 %Hb that are : Values of 5.6 %Hb that are erroneously measured as 6.5 %Hb or highererroneously measured as 6.5 %Hb or higher
2626
ADA Diagnostic Criterion vs NGSP vs CAP
2727
ADA treatment goal 7.0 %HbADA treatment goal 7.0 %Hb
When reWhen re--evaluate or change treatment?evaluate or change treatment?
Earlier ADA recommendation was to Earlier ADA recommendation was to adjust treatment if HbA1c 8.0 %Hb or adjust treatment if HbA1c 8.0 %Hb or greatergreater
Change of 1.0 %Hb is equivalent to a Change of 1.0 %Hb is equivalent to a monitoring requirement of about 14%monitoring requirement of about 14%
Absent from latest guidelinesAbsent from latest guidelines
HbA1c Requirement for HbA1c Requirement for Monitoring TreatmentMonitoring Treatment
2828
Diagnosis & Monitoring (Δ1.0%Hb) vs NGSP vs CAP PT
2929
What quality is needed in a What quality is needed in a medical laboratory for the medical laboratory for the
intended clinical use of HbA1c?intended clinical use of HbA1c?
CAP criterion was 10% in 2009, 8% in CAP criterion was 10% in 2009, 8% in 2010 (8%) and will be 6% in 20112010 (8%) and will be 6% in 2011
NGSP 2010 criterion of NGSP 2010 criterion of ±±0.75 %Hb 0.75 %Hb corresponds to TEa of 10.7% @ 7.0 %Hb;corresponds to TEa of 10.7% @ 7.0 %Hb;
CLIA criterion for acceptable performance CLIA criterion for acceptable performance of glucose is 10%of glucose is 10%
3030
Example ApplicationExample ApplicationMethod Performance Characteristics and QC Design
Test (Units) Glycated Hemoglobin (%Hb)Method (Analyzer) DCA Vantage
Medical Decision LevelsConcentrations
LOW Xc MID Xc High Xc5.0 7.0 9.0
CLIA Quality Criterion %TEa 10% 10% 10%
Precision (Replication or QC Data)SD
Mean%CV
Bias (Comparison, PT, Peer Data)Calculated Bias
%Bias
Sigma‐Metric(%TEa)/%CV
(%TEa ‐
%Bias)/%CV
SQC from Sigma tool, Control RulesTotal Number Measurements, NAnalytical QC Strategy
3131
Precision results Precision results from Lenters Studyfrom Lenters Study
Lenters-Westra E, Slingerland RJ. Six of Eight Hemoglobin A1c Point-of-Care Instruments Do Not Meet the General Accepted Analytical Performance Criteria. Clin Chem 2010;56:44-52.
3232
Example ApplicationExample ApplicationMethod Performance Characteristics and QC Design
Test (Units) Glycated Hemoglobin (%Hb)Method (Analyzer) DCA Vantage
Medical Decision LevelsConcentrations
LOW Xc MID Xc High Xc5.0 7.0 9.0
CLIA Quality Criterion %TEa 10% 10% 10%
Precision (Replication or QC Data)SD
Mean 5.1%Hb 11.2%Hb%CV 1.8% 3.7%
Bias (Comparison, PT, Peer Data)Calculated Bias
%Bias
Sigma‐Metric(%TEa)/%CV
(%TEa ‐
%Bias)/%CV
SQC from Sigma tool, Control RulesTotal Number Measurements, NAnalytical QC Strategy
3333
Accuracy results Accuracy results --
Comparison Comparison with avg of 3 reference methodswith avg of 3 reference methods
3434
What do the statistics tell us What do the statistics tell us about the size of errors?about the size of errors?
Example ApplicationExample ApplicationMethod Performance Characteristics and QC Design
Test (Units) Glycated Hemoglobin (%Hb)Method (Analyzer) DCA Vantage
Medical Decision LevelsConcentrations
LOW Xc MID Xc High Xc5.0 7.0 9.0
CLIA Quality Criterion %TEa 10% 10% 10%
Precision (Replication or QC Data)SD
Mean 5.1%Hb 11.2%Hb%CV 1.8% ~2.75% 3.7%
Bias (Comparison, PT, Peer Data)Calculated Bias
%Bias 1.3‐3.0%
Sigma‐Metric(%TEa)/%CV
(%TEa ‐
%Bias)/%CV 3.16‐2.55SQC from Sigma tool, Control Rules Multi‐rule QCTotal Number Measurements, N All the QC you can afford!Analytical QC Strategy Hope and pray nothing goes wrong!
4040
Houston Houston ––
We have a problem!We have a problem! Or maybe not???Or maybe not???
Waived test doesnWaived test doesn’’t have to meet US t have to meet US CLIA minimum QC of 2/day, nor EQC of CLIA minimum QC of 2/day, nor EQC of 2/week or 2/month!2/week or 2/month!
Just follow manufacturerJust follow manufacturer’’s instructions s instructions and recommendationsand recommendations
Not required to validate method Not required to validate method performance!performance!
Not required to participate in PT!Not required to participate in PT!4141
WhatWhat’’s the point?s the point?
11stst
issue is whether the device can achieve issue is whether the device can achieve
the necessary quality when it is working the necessary quality when it is working correctly?correctly?
Method validation is critical, but not required!Method validation is critical, but not required!
Quality is controlled by FDA when device is Quality is controlled by FDA when device is approved as approved as ““waivedwaived””
22ndnd
issue is that QC only monitors the issue is that QC only monitors the
Of little use if device canOf little use if device can’’t achieve desired t achieve desired performanceperformance
4242
Is quality acceptable for POC Is quality acceptable for POC application?application?
POC application POC application lessless demanding because demanding because the test should be used only for the test should be used only for monitoring, not diagnosis!monitoring, not diagnosis!
ADA guidelines recommend against using POC ADA guidelines recommend against using POC devices for diagnosis of diabetesdevices for diagnosis of diabetes
For monitoring therapy, what changes need to For monitoring therapy, what changes need to be detected? be detected?
See discussions on See discussions on www.westgard.comwww.westgard.com4343
Huge effort needed to harmonize Huge effort needed to harmonize laboratory tests when have many different laboratory tests when have many different measurement principles and proceduresmeasurement principles and procedures
HbA1c illustrates the complexity, as well as HbA1c illustrates the complexity, as well as the difficulty in achieving comparabilitythe difficulty in achieving comparability
Certification of equivalent results only Certification of equivalent results only indicates comparability indicates comparability ““on average,on average,””
not that not that
individual patient results will be comparableindividual patient results will be comparable
““ProprietaryProprietary””
methodology limits efforts for methodology limits efforts for standardizing measurement proceduresstandardizing measurement procedures
Global and national efforts to harmonize Global and national efforts to harmonize HbA1c testing have improved quality, HbA1c testing have improved quality, BUT BUT biases still exist between different biases still exist between different measurement principles and procedures.measurement principles and procedures.
Still have problems with practices for Still have problems with practices for defining clinical treatment guidelines and defining clinical treatment guidelines and setting method operating specifications setting method operating specifications that can assure attainment of intended that can assure attainment of intended quality of test results.quality of test results.
4747
Further Complications Today! ISO Measurement Uncertainty
If measurement uncertainty is to be the measure of quality, the assumption is that all biases can be eliminated or corrected!
Not true for today’s proprietary methods!
Must “harmonize”
metrological theory with practical tools and techniques from current error management framework!
Medical laboratories should place priority on traceability, not measurement uncertainty!
4848
How explain this to customers who How explain this to customers who believe all methods give same results?believe all methods give same results?
4949
In conclusion, a hydrangea is a In conclusion, a hydrangea is a hydrangea is a hydrangea! hydrangea is a hydrangea!
5050
Harmonized, but not always the Harmonized, but not always the same!same!