FSIS Guidance for Evaluating Test Kit Performance 10/15/10 Page 1 of 22 FSIS Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods Table of Contents I. Introduction II. General Guidance for Evaluation of Pathogen Test Kit Performance 1. Purpose, Scope and Audience 2. General Considerations 3. Inoculum - Number of strains - Strain selection (Table 1 ) - Preparation of inoculum - Inoculation of matrix 4. Matrix - Choice of matrix (Tables 2a , Table 2b ) - Decision Criteria Illustrating When New Validation Studies Should be Conducted for an Existing Method - Characterization of matrix 5. Study Design and Analysis - Paired and Unpaired studies (Figure 1 ) - Data Analysis - Levels of validation (Figure 2 , Table 3 ) - Reference method 6. Sample size - Unpaired studies (Figure 3 ) - Paired studies (Figure 4 ) 7. Study report Attachment. Example of Pearson Chi-square statistic Statistic Calculation for Unpaired Samples
22
Embed
Compliance Guidelines: Conducting Validation Studies for
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
FSIS Guidance for Evaluating Test Kit Performance 10/15/10
Page 1 of 22
FSIS Guidance for Test Kit Manufacturers, Laboratories:
Evaluating the Performance of Pathogen Test Kit Methods
Table of Contents
I. Introduction
II. General Guidance for Evaluation of Pathogen Test Kit Performance
1. Purpose, Scope and Audience
2. General Considerations
3. Inoculum
- Number of strains
- Strain selection (Table 1)
- Preparation of inoculum
- Inoculation of matrix
4. Matrix
- Choice of matrix (Tables 2a, Table 2b)
- Decision Criteria Illustrating When New Validation Studies Should be Conducted
for an Existing Method
- Characterization of matrix
5. Study Design and Analysis
- Paired and Unpaired studies (Figure 1)
- Data Analysis
- Levels of validation (Figure 2, Table 3)
- Reference method
6. Sample size
- Unpaired studies (Figure 3)
- Paired studies (Figure 4)
7. Study report
Attachment. Example of Pearson Chi-square statistic Statistic Calculation for Unpaired
Samples
FSIS Guidance for Evaluating Test Kit Performance 10/15/10
Page 2 of 22
I. Introduction
FSIS-regulated establishments rely on results from pathogen testing programs to comply
with regulatory requirements and to support decisions made in their HACCP systems to
ensure the production of safe unadulterated products. FSIS does not maintain a list of
acceptable methods to be used in these testing programs. However, the Agency’s overall
expectation is that any test used by an establishment is appropriate for its intended use,
that the test performance is comparable to the FSIS method (if applicable), and that the
laboratory performing the test did not introduce modifications that could compromise
test’s performance.
FSIS believes that a robust validation study must be performed on any method used by
establishments to detect microbiological hazards in FSIS-regulated foods. A validation
study is an experimental process to measure performance characteristics of a particular
test, with the goal of determining whether the test is equivalent to the reference test.
“Equivalent” is defined as the designated relationship between two tests indicating that,
for the intended conditions of use, the performance characteristics are statistically
indistinguishable.
This guidance document (section II) provides an example of how to design a robust
pathogen method validation study that may be used to demonstrate equivalence. The
performance characteristics addressed in this guidance are described in Box 1. The
guidance provided in this document primarily focuses on measuring relative recovery and
sensitivity (false negative rate). Measurement of specificity (false positive rate),
inclusivity, exclusivity, repeatability, reproducibility, and ruggedness should be
performed through the direction of an independent organization, or by following
guidance provided by the AOAC International Official Methods of Analysis Program2.
The FSIS guidance should be useful to organizations that design or conduct validation
studies for foodborne pathogen testing methods. These organizations include test kit
manufacturers, laboratories, and independent validation organizations.
The FSIS guidance is not intended to conflict with or supplant existing guidance from
independent organizations (AOAC and ISO).
The FSIS guidance could be used to evaluate the performance of a candidate alternative
method for Escherichia coli O157:H7 or non-O157 Shiga toxin producing E. coli (non-
O157 STEC).
NOTE: The use of validation in this document is not intended to have any application to
the implementation of 9 CFR 417.4(a)(1) on initial validation of HACCP plans. This
document deals exclusively with the evaluation of pathogen test kit methods.
FSIS Guidance for Evaluating Test Kit Performance 10/15/10
Page 3 of 22
Box 1. Performance Characteristics Used to Evaluate Pathogen Test Kit Methods
Relative Recovery measures the proportion of true positive samples recovered from the
new test compared to the reference test when similarly inoculated.
Sensitivity/False Negative Rate measures the probability that a test will correctly detect a
true positive sample. A false negative (FN) result occurs when a test does not correctly
detect a true positive sample, so 1 minus the sensitivity equals the FN rate.
Specificity1/False Positive Rate measures the probability that a test will correctly detect a
true negative sample. A false positive (FP) result occurs when a test does not correctly
detect a true negative sample, so 1 minus the specificity equals the FP rate.
Inclusivity measures the ability of a test to detect a wide variety of strains representing
the target pathogen.
Exclusivity measures the ability of a test to resist interference by cross-reactivity with
non-target organisms likely to be found in the tested food.
Reproducibility is a measure of test performance in different laboratories with different
equipment and personnel.
Repeatability is a measure of test performance in the same laboratory with the same
equipment and personnel.
Ruggedness testing is performed to determine if small changes to the procedure or
environmental factors influences test performance.
Validation studies are designed to evaluate the performance of a new test (referred in this
document as the alternative method, or A) against a reference method (referred to as R)
that provides a definitive result. The typical study design can not be applied to methods
which do not have an available authoritative R. Additionally, the study design described
in this document (section II) would not determine if the performance of A exceeded R.
Validation studies performed through the Association of Analytical Communities
(AOAC) or other recognized independent organizations that perform or organize
validation studies on behalf of test developers, follow the traditional design and rely on
culture based reference methods. FSIS believes that any method used to detect foodborne
pathogens in meat, poultry, and egg products should be as sensitive as the FSIS method.
Other recognized cultural methods, fit for the purpose of detecting low levels of stressed
cells in food, also may be an appropriate reference method. Alternative methods should
be re-validated when significant changes affecting performance are introduced to the
reference method. Re-validation should be performed within one year of the introduced
changes.
1 Also referred to as Selectivity.
FSIS Guidance for Evaluating Test Kit Performance 10/15/10
Page 4 of 22
From a food safety perspective, methods to detect foodborne pathogens should be
validated using a robust study design with special attention to sensitivity (false negative
rate) and inclusivity to limit or prevent false negative results. From an economic
perspective, methods should be validated with special attention to specificity and
exclusivity to prevent or limit false positive results and to reduce the time to obtain
results, thus allowing product disposition to be rapidly determined. Sensitivity, specificity
and timeliness are related, so an increase in one parameter may lead to a reduction of the
other.
Robust validation methodology implies that A should be evaluated under the most
challenging conditions to provide confidence that the method likely will perform well
under most situations. In practical terms, a robust validation methodology should address
the following parameters:
1. The inoculum level should be low enough to achieve fractional recovery of positive
results by R. In FSIS’ experience, pathogens subjected to zero tolerance testing in
meat, poultry, and egg products often are found at low levels, close to one viable
organism per analytical unit. Because it is practically impossible to place a single
organism in a testable unit of food, the best approach taken in validation studies is to
inoculate foods at low levels so that a fraction of the analyses (defined as 20-80% of
the inoculated samples analyzed by R) are confirmed positive for the target pathogen.
“Fractional recovery” is a well-established concept used by AOAC and other
organizations performing validations, and was recognized as a preferred method for
defining test performance by the Presidential Task Force for Best Practices in
Microbiology2,3
.
2. The study should evaluate the ability of the test to detect potentially stressed or
injured cells. Foods prepared for commercial distribution often are exposed to
conditions injurious to bacterial contaminants. Foods often are processed at reduced
temperatures to prevent pathogen growth and avoid spoilage. In other situations, food
properties are modified by the application of antimicrobial agents such as organic
acids, salt, curing agents, or other preservatives or by the modification of pH and
water activity. In addition, the presence and level of resident microflora in the sample,
(which is related to the age and handling of the product sample) could interfere with
target pathogen growth. Any of these treatments may negatively affect the growth
properties of the target pathogen, by extending lag phase or exponential growth rate.
Injured cells would be more difficult to detect, but could retain their ability to cause
illness.
3. The study should evaluate the ability of the test to detect target organisms in the
products likely to be tested; however, foods that present a challenge to the test’s
performance should be evaluated, even if they are not as likely to be tested.
4. The study should evaluate a target strain with limited growth potential in the product
to be tested; this would present a challenge to the sensitivity of the test kit.
2 Feldsine et al., AOAC INTERNATIONAL Methods Committee Guidelines for Validation of Qualitative
and Quantitative Food Microbiological Official Methods of Analysis. Journal of the AOAC International
85(2): 1187-1200. 3AOAC International Presidential Task Force, Best Practices in Microbiological Methodology (August 10,