1 © 2012 3M. All Rights Reserved. METHOD VALIDATION & VERIFICATION APEC-PTIN Project: CHILE 2013
1 © 2012 3M. All Rights Reserved.
METHOD VALIDATION & VERIFICATION APEC-PTIN Project:
CHILE 2013
2 © 2012 3M. All Rights Reserved.
§ Provide definitions of validation & verification from key resources § List some key performance characteristics for method validation § Provide an overview of 3rd party method validation schemes § Distinguish validation and verification § List some key performance characteristics for method verification § Distinguish verification: method verification vs “continuous” verification § Provide links to references for further use / information
OBJECTIVES
3 © 2012 3M. All Rights Reserved.
§ CODEX - CAC/GL 74-2010: Guidelines on performance criteria and validation of methods for detection, identification and quantification of specific DNA sequences and specific proteins in foods
§ ISO/DIS 16140-1: Microbiology of food and animal feed, Method validation, Part 1: Vocabulary Currently under revision
§ USDA FSIS: Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods; and Establishment Guidance for the Selection of a Commercial or Private Microbiological Testing Laboratory
§ US FDA: Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods
§ Health Canada: The Compendium of Analytical Methods, Volume 1, Development of Methods
§ ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories
Main Validation & Verification Definition Sources:
4 © 2012 3M. All Rights Reserved.
Reference Validation Definition Codex CAC/GL 741
Process to establish the performance characteristics and limitations of an analytical method: which analytes, in what kind of matrices, in the presence of which interference. Result = precision and trueness values of a certain analytical method under the examined conditions.1 [pg 2]
ISO 16140-12 Establishment of the performance characteristics of a method and provision of objective evidence that the performance requirements for a specified intended use are fulfilled.2 [pg 2]
USDA FSIS3 Process to measure performance characteristics of a particular test, with the goal of determining whether the test is equivalent to the reference test for the intended conditions of use. “Equivalent” = the performance characteristics are statistically indistinguishable.3 [pg 2]
US FDA4 Demonstration that adequate confidence is provided when the results obtained by the alternative method i.e. the commercially available kit, are comparable to or exceed those obtained using the reference method using the statistical criteria contained in the approved validation protocol.4 [pg 16]
Health Canada5 Evaluation of the performance parameters of a new method in comparison to an accepted reference method using paired or unpaired samples. In the context of relative validation, the results of the reference method are assumed to reflect the true microbiological status of the samples and the performance parameters of the alternative method are calculated in relation to this. (Source: Part 4)
ISO 17025:20056 The confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled.6
WHAT is Validation?
5 © 2012 3M. All Rights Reserved.
Validation: 1. Performance characteristics are defined, then… 2. compared to a reference method, and then… 3. statistically evaluated to determine “equivalence”
A method is validated for the matrices [& sample sizes] included in the validation
Summary of ALL Validation reference definitions:
6 © 2012 3M. All Rights Reserved.
Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study1
Performance Characteristic Quantitative Qualitative (presence/absence)
Relative Accuracy (% Recovery) Yes No
Matrix Effects Yes Yes
Precision Yes No
Sensitivity Yes Yes
Specificity Yes Yes
Inclusivity Yes Yes
Exclusivity Yes Yes
False Positive Rate Yes Yes
False Negative Rate Yes Yes
LOD [POD or LOD50] Yes Yes
LOQ Yes No
Ruggedness Yes Yes
Linearity/Range Yes No
Dr. Michael Brodsky AOAC International 2013 Annual Meeting
7 © 2012 3M. All Rights Reserved.
Methods requiring validation: ― New or original methods ― Modifications to validated methods ― Extension of scope to include additional analytes, matrices, or changes in
intended use ― Changes involving new technology or automation ― Significant parameter changes: reagents/apparatus, time /temperature
incubation periods, enrichment media, etc.
It is possible that small changes/matrix extensions to a related product etc would not
need a ‘full validation’ but may need some level of testing to assess for unintended differences.
8 © 2012 3M. All Rights Reserved.
§ “Validated” methods are required for conducting “Official Testing” ― Official = adjective, relating to an authority or public body and its activities and
responsibilities [Oxford online dictionary]
§ European Union regulations specifically stipulate validated methods that have been collaboratively studied
§ Internationally, regulatory agencies require, or at least prefer, to use methods that have been validated and collaboratively studied.
§ Some global regulatory agencies require AOAC Official Methods of Analysis to receive Final Action, before they may be used as an “Official” method
WHY are Validated methods important?
9 © 2012 3M. All Rights Reserved.
Agency Method Validation Requirements Codex CAC/GL 218
5.2.1 Whenever possible, only methods for which the reliability (accuracy, reproducibility, inter- and intra-laboratory variation) has been statistically established in comparative or collaborative studies in several laboratories should be used.
(EC) No 2073/20079
The use of alternative analytical methods is acceptable when the methods are validated against the reference method in Annex I and if a proprietary method, certified by a third party in accordance with the protocol set out in EN/ISO standard 16140 or other internationally accepted similar protocols, is used. Pg 9
USDA FSIS11 All analytical methods described in FSIS-MLG have been scientifically validated and are considered fit for their intended purpose. Laboratories may also use other validated testing methods that differ from the methods described in FSIS-MLG. For the intended conditions of use, the performance characteristics of the new method and the well characterized method should be statistically indistinguishable Pg 13
US FDA4 FDA-BAM presents the agency's preferred laboratory procedures for microbiological analyses of foods and cosmetics. Methods…fully validated in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc. require in-house verification for the “first use”…
Health Canada5 HPB Methods and Laboratory Procedures are generally used to determine compliance with the Food and Drugs Act and Regulations. In general, methods published in the Compendium of Analytical Methods for which the "application" section is appropriate for the intended purpose are used for microbiological testing of food.
REQUIREMENTS of methods for “Official” use
10 © 2012 3M. All Rights Reserved.
§ Official Methods = Methods used or accepted for use by a regulatory authority ― Traditional, cultural methods [may be jurisdiction specific] ― ISO Method ― ISO 16140 Validated Methods ― AOAC Official Methods ― FDA Bacteriological Analytical Methods ― USDA FSIS Microbiology Laboratory Methods ― HC Compendium of Analytical Methods, Volumes 1 to 5
§ Any of the above methods MAY be accepted for use by regulatory agencies, IF: • They have been compared to an acceptable reference method • They have been validated as “fit for purpose” for the matrices analyzed
What is considered an OFFICIAL method?
11 © 2012 3M. All Rights Reserved.
§ ISO ― AFNOR: French National Organization for Standardization ― MicroVal: European certification body for microbiology methods ― NordVal: Nordic certification body: Denmark, Finland, Iceland, Norway and Sweden
§ AOAC Research Institute [RI]: International Standards Organization ― Performance Tested Method (PTM) ― Official Method of Analysis (OMA)
§ Health Canada: Canadian certification through Microbiology Methods Committee (MMC) ― Official Methods (MFOs) ― HPB Methods (MFHPBs) ― Laboratory Procedures (MFLPs)
Validation Schemes
12 © 2012 3M. All Rights Reserved.
AFNOR and MicroVal Certification schemes based on EN/ISO 16140: § Comparison Study to reference method
1. Independent expert laboratory co-ordinates validation
2. Inter-laboratory study [collaborative study]
3. “All Foods” Claim
§ Regular audit of the quality system of the manufacturer [based on EN 29002 standard (ISO 9002)]
§ Regular verification of the certified methods post certification [every 4 years]
13 © 2012 3M. All Rights Reserved.
AOAC Research Institute § AOAC Performance Tested Method [PTM]
― Pre-collaborative + 1 Independent lab: I/E; Ruggedness testing, paper review of quality system.
― Yearly renewal – paper process only
― NO all foods claim – only for matrices evaluated
§ AOAC PTM / Official Method of Analysis [OMA] Harmonization ― Complete AOAC PTM program / Initiate collaborative study to achieve OMA First Action status;
Final Action status after 2 yrs use in the industry without issue
§ AOAC PTM / OMA/ MicroVal Harmonization
― Both programs run simultaneously – 1 collaborative study
14 © 2012 3M. All Rights Reserved.
§ Micro Food Lab Procedure [MFLP] ― Independent lab [pre-collaborative study] ― All Foods Claim ― Reviewed every 5 years ― Paper review of quality system
§ Health Protection Food Branch [MFHPB] Methods
― Must be an MFLP method for 1 year prior to collaborative study
Health Canada
15 © 2012 3M. All Rights Reserved.
§ AFNOR http://www.afnor-validation.com/afnor-validation-validated-methods/validated-methods.html
§ MicroVal http://www.microval.org/validated-methods.html
§ AOAC http://www.aoac.org/iMIS15_Prod/AOAC_Member/Research_Institute/RI_Main.aspx?WebsiteKey=2e25ab5a-1f6d-4d78-a498-19b9763d11b4&hkey=33b744f6-f71e-456a-9305-552469587666&CCO=5
§ Health Canada http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/index-eng.php (Volumes 1 to 5)
§ NordVal http://www.nmkl.org/NordVal/NordValMetoder.htm
Method Scheme Websites – list validated methods including claimed matrices, sample sizes
16 © 2012 3M. All Rights Reserved.
§ A global Stakeholder panel was formed [through AOAC RI]
§ Performance characteristics between agencies/schemes were compared
§ 5 areas for harmonization: 1. Reference methods – so much variation, some matrix specific 2. Definition of Fractional Positives – 25-75% positive out of 20 samples 3. Selection of Food/Category – Chart created for incorporation into ISO 16140-2 4. Statistical Analysis – RLOD, Chi-Square and/or POD models 5. Data Sets for Qualitative Collaborative Study/Sample Size – 10 labs / 25 g samples
§ Recent addition: “All Foods” claim now = “Broad range of foods”
Harmonization – ISPAM International Stakeholder Panel on Analytical Methods
17 © 2012 3M. All Rights Reserved.
Reference Verification Definition Codex Could not find a method verification definition Codex documents [only process verification]
ISO 16140-1 Demonstration that a validated method functions in the user's hands according to the method's specifications determined in the validation study and is fit for purpose
USDA FSIS Laboratory should demonstrate on-going competence in performing the method at their facility, which would include participating in proficiency test programs11 Pg 13
US FDA The confirmation by examination and the provision of objective evidence that specified requirements have been fulfilled.4 Pg 16
Health Canada Verification studies are required when a laboratory implements a fully validated, published or reference standard method. The laboratory should provide evidence, by conducting a secondary evaluation study, that it is capable of achieving the published performance characteristics of the method.5 Vol 1; Part 4; 3.4
ISO 17025:2005 The Laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations
Comparison of VERIFICATION definitions
18 © 2012 3M. All Rights Reserved.
§ Proof that the laboratory can perform the [validated] method as intended
§ FDA has detailed Guidance on in-house verification of a validated method
before use4 Pg 16
Verification for USE
This applies only to matrices and sample sizes that were included in the validation study
19 © 2012 3M. All Rights Reserved.
§ Validation ― Demonstrate the METHOD is “equivalent” to the reference method [for the
matrices validated] based on defined method criteria
§ Verification ― Demonstrate in the hands of the USER that the method can be performed
to meet the defined method criteria
Distinguishing Validation and Verification
20 © 2012 3M. All Rights Reserved.
Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method1
Performance Characteristic VALIDATION Quantitative
VALIDATION Qualitative (P/A)
VERIFICATION (Where Applicable)
Relative Accuracy (% Recovery) Yes No Yes Matrix Effects Yes Yes Yes
Precision Yes No Yes
Sensitivity Yes Yes Yes
Specificity Yes Yes Yes
Inclusivity Yes Yes No
Exclusivity Yes Yes No
False Positive Rate Yes Yes Yes
False Negative Rate Yes Yes Yes
LOD [POD or LOD50] Yes Yes No
LOQ Yes No No
Ruggedness Yes Yes No
Linearity/Range Yes No No
Dr. Michael Brodsky AOAC International 2013 Annual Meeting
21 © 2012 3M. All Rights Reserved.
§ Proficiency Testing: ― Should be done several times per year ― Should be handled as a routine sample – not specially handled ― Can help determine the performance of individual laboratories for specific tests ― Is used to monitor laboratories’ continuing performance ― Can demonstrate an analyst’s ability to properly perform the test
§ Test samples: ― Certified proficiency test providers ― Internal Ring tests
CONTINOUS Verification - ISO 17025 requirement
22 © 2012 3M. All Rights Reserved.
§ Definitions of validation & verification § Key performance characteristics for method validation and verification § Several global 3rd party method validation schemes were outlined § Distinguish validation and verification § References for further use / information
SUMMARY
23 © 2012 3M. All Rights Reserved.
24 © 2012 3M. All Rights Reserved.
1. CODEX - CAC/GL 74-2010 Guidelines on performance criteria and validation of methods for detection, identification and quantification of specific dna sequences and specific proteins in foods http://www.codexalimentarius.org/standards/list-of-standards/
2. ISO/DIS 16140-1 Microbiology of food and animal feed — Method validation — Part 1: Vocabulary Currently under revision 3. USDA FSIS: Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods
http://www.fsis.usda.gov/shared/PDF/Validation_Studies_Pathogen_Detection_Methods.pdf 4. US FDA: Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods
http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf; 5. Health Canada: Procedure for the Development and Management of Food Microbiological Methods; Part 1. Definitions and Explanations of
Methods and General Procedures; 3. Definition of Terms, 3.1 Method http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/method_development-eng.php
6. ISO 17025:2005 General requirements for the competence of testing and calibration laboratories NO LINK, this is copyrighted material 7. 2013 AOAC International Annual Meeting presentation, with permission from: Michael Brodsky, President, Brodsky Consultants, Thornhill, Ontario,
Canada [email protected] 8. CODEX – CAC/GL 21- 1997 Principles for the Establishment and Application of Microbiological Criteria for Foods
http://www.codexalimentarius.org/standards/list-of-standards/ 9. COMMISSION REGULATION (EC) No 2073/2005 of 15 November 2005 on microbiological criteria for foodstuffs :
http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2005:338:0001:0026:EN:PDF 10. Health Products and Food Branch (HPFB) Standards and Guidelines for Microbiological Safety of Food - An Interpretive Summary 6. Equivalent
Methods http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/intsum-somexp-eng.php 11. USDA FSIS: Establishment Guidance for the Selection of a Commercial or Private Microbiological Testing Laboratory:
http://www.fsis.usda.gov/shared/PDF/Guidance_Selecting_Micro_Testing_Lab.pdf 12. AOAC: AOAC INTERNATIONAL Methods Committee Guidelines for Validation of Microbiological Methods for Food and Environmental Surfaces
http://www.aoac.org/imis15_prod/AOAC_Docs/StandardsDevelopment/AOAC_Validation_Guidelines_for_Food_Microbiology-Prepub_version.pdf
13. Official Methods for the Microbiological Analysis of Foods http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/index-eng.php
References