Method Validation and Verification: An Overview Patricia Hanson, Biological Administrator I Florida Department of Agriculture and Consumer Services, Food.

Method Validation and Verification: An OverviewPatricia Hanson, Biological Administrator I

Florida Department of Agriculture and Consumer Services, Food Safety Microbiology

Laboratory

Introduction• 17 years in the microbiology section of the

Florida Department of Agriculture and Consumer Services, Food Laboratory

• Laboratory moved from primarily cultural methods in 1998 to a leader in the implementation of new technology in 2015

• Numerous partnerships with state, federal and industrial customers to validate and implement new methods

Goals• Distinguish between method verification and

method validation• Based on our laboratory’s experience, identify

the critical components of both a method verification and a method validation

• Offer insight to approaches we have found useful, specifically in the microbiology section

• Share lessons learned

Verification vs Validation• Verification is a demonstration that your

laboratory can perform a Standard Method or other well documented method and produce acceptable results

• Validation is both a demonstration that your lab can run the method and that the method or change to a method is fit for purpose

Aspects of a Verification• Run entire method, start to finish, exactly as it

is written• Include both positive and negative samples• Represent all matrix types• Be spiked at a level that is fit for the purpose

of the method if spiked samples are used

Verifications can be ongoing….• For the purpose of this discussion,

“Verification” is being used to describe what the laboratory does prior to implementing a method in the laboratory

• The term “Verification” is often also used to describe the ongoing process of verifying a method with each run – in our lab this it the use of Process Controls, Cultural Controls, and Media Controls

Validation• Laboratory developed method

• Matrix Extensions

• ANY technically significant change to a Standard Method or previously validated method

Overview of an Existing Guideline

• Level One: Urgent Use, Single Lab Validation

• Level Two: Single Lab Validation

• Level Three: Multi-Lab Validation

• Level Four: Full Collaborative Study

Level 2: Single Lab Validation Highlights

• 50 different strains of target organism, if available

• 30 different strains of “nearest neighbor” organisms

• At least two target analyte levels

• 20 replicates per food type per analyte level

• Perform side by side if possible

Aspect of a Verification• Be extensive enough to show that the method

is fit for the intended use and meets the customer’s needs

What is “Extensive Enough”Items to be considered:Matrix: number and typesTarget: source and levelsMethod Evaluation: side by side or use of

standard reference material When considering these items during the

development of the validation, keep in mind what is needed to show the method is fit for the intended use and meets the customer needs.

Matrices• Validation or verification needs to be

performed on each food type• Meat (raw, smoked, cooked, etc)• Vegetable (whole, cut, sprout, etc)• Dairy (liquid, solid, etc)• Other (spices, candies, etc)

Replicates• Exact number depends on method• Verification, normally triplicate• Validation, normally six replicates per level• Important to do the same number of

replicates of “un-spiked”

Naturally Contaminated Samples• Naturally contaminated pros and cons

• Best test of the method

• Level of analyte is unknown

• Little or no control of analyte level

Spiked Samples• Spiked sample pros and cons• Ability to control organism level• Methods often perform differently on spiked

samples than on naturally contaminated samples

Aging of Spiked Cultures• Spiking samples 48-72 hours prior to initiation

of testing can better represent naturally occurring contamination

• Sometimes the cells are so stressed, they die off

• Consider the method or part of method being validated and type of validation

Selection of Organisms• Select at least one organism for each target

• “Nearest Neighbor” organisms – not often used but good to consider

“Side by Side” Validations• Normally involves running the method being

validated side by side with a Standard or established method

• Twice the work but very valuable for determining if the “new” method is equivalent or better than the Standard or established method

• Best choice when changing part of a method – for example a new enrichment broth but same screening method

How Much Confirmation?• Screening Method Validations: Confirm

everything• You can get by with an abbreviated

confirmation, for example just biochemical panel and/or serology but do enough to show you “got out what you put in.”

Why Do So Many Confirmations?• Ensure you can confirm samples that screen

positive

• Ensure samples you can confirm screen positive

• Spiked samples that don’t screen positive and don’t confirm point to limitations of the method or problems with the spiking procedure

Data Evaluation• The measures used to evaluate data depend on

the purpose and type of the method and the needs of the customer

Accuracy

RUGGEDNESSLimit

of

Quantitatio

n

SPECIFICITY

False Positive RatePrecision

Range

Qualitative Data Evaluation• Detection Limits – can the method detect the target

at the levels required by the customer?• Specificity – can the method detect the target in the

matrix?• Selectivity – can the method differentiate between

the target and other analytes in the matrix?• Robustness – can the method be repeated on the

same sample with the same results?

Quantitative Data Evaluation• Detection Limits, Specificity, Selectivity, and

Robustness• Precision – Does the method produce the

same results as established method?• Linearity – Does the method produce

proportional results with in the customer’s target range?

Personnel Considerations• Personnel designing and approving verifications

and validations should be knowledgeable about the technology as well as the needs of the lab

Personnel Considerations• Personnel carrying out verification and

validations should be competent in all techniques

• Work on verification and validation studies can be used to demonstrate competency but should not be used as training

Pit-falls of Method Validation• Repeating tests until you get the result you

want • Drifting from method validation into method

development• Adjusting criteria for acceptance AFTER the data is obtained

Overcoming Pit-falls• Have a well thought out, documented and

approved plan and stick to it

• Go back and start over, incorporating any changes into new validation

• All data should still be included in evaluation

Documentation

• Approval of plan prior to running• Raw Data• Interpretation of raw data • Final approval from customer• Reference validation in SOP• If SOP is a modified Standard Method, list

deviations from the Standard Method in a section in the SOP

References• U. S. Department of Health and Human Services, Food and

Drug Administration (2014, August 29). Volume II – Methods, Method Verification and Validation ORA-LAB.5.4.5. Retrieved from http://www.fda.gov/ScienceResearch/FieldScience/ucm171877.htm

• U. S. Department of Health and Human Services, Food and Drug Administration (2011, September 8). Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods. Retrieved from http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf

Questions