Handling OOS, OOT and Unexpected Results Karen Ginsbury PCI Pharmaceutical Consulting Israel Ltd for IFF, October 2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Handling OOS, OOT and
Unexpected Results
Karen Ginsbury PCI Pharmaceutical Consulting Israel Ltd for IFF, October 2017
What is an OOS
What is an OOT
What is an Unusual, Questionable, Atypical Result
Objective: of this course…
and BE objective when assessing OOS, OOT
Understand what an OOS, OOT, Unusual result is
Understand why Data Integrity is a BURNING hot issue
Understand what you can and can’t do and WHY
5
There is no place for emotions – put them aside
Write Down Keywords relating to OOS
Review and Comment
KG Review
Warning Letter 19 October, 2016 China
You do NOT want to go there…
You do NOT want to go there…
You do NOT want to go there…
Unofficial Testing
b. Your firm frequently performs “unofficial testing” of samples, disregards the results, and reports results from
additional tests. For example, during stability testing, your firm tested a batch sample six times and subsequently
deleted this data.
You performed “trial” sample HPLC analyses prior to acquiring the “official” analyses. The “trial” results were
subsequently discarded. “Trial” HPLC analyses were apparently run as part of the 12-month long-term stability
studies on batch #15069 for related substances. Your employee ran an HPLC analysis sequence and subsequently deleted the raw data files. Your quality control staff named the samples using the last three digits
of the batch numbers to link the "trial" injections with the official assay analyses. Your Senior Quality Control (QC)
Officer confirmed these were analyses of batch samples. Furthermore, we found that this batch was analyzed
for unknown impurities and results reported as within specifications. However, the data showed that the "trial"
injection for this batch failed the unknown impurities specification in several test runs.
Your Senior QC Officer confirmed that QC laboratory employees had frequently practiced the use of “trial”
injections at your facility. Significantly, in addition to the example above, our inspection found 5,301 deleted
chromatograms on a computer used to operate two HLPC instruments in your QC laboratory. Many of these
files were “trial” injections of batches.
Important Disclaimer
Karen Ginsbury is a consultant
Your company has approved SOPs and a quality system
Anything that Karen says is for consideration to stimulate thought and discussion and may
result in review or revision of SOPs and procedures.
AS LONG AS SOPs ARE NOT UPDATED under change control, you and your company
PERSONNEL FOLLOW THE APPROVED PROCEDURE
“Karen said” is NOT an excuse!
14
US CGMP’s
21 CFR 211.192
“Any unexplained discrepancy of the failure of a batch or any of its contents to meet any
of its specifications shall be thoroughly investigated, whether or not the batch has already
been distributed.”
“The investigation shall extend to other batches of the same drug product and other drug
products that may have been associated with the specific failure or discrepancy.”
“A written record of the investigation shall be made and shall include the conclusions and
follow-up.”
15
EU GMP’s Chapter 6: Quality Control
Laboratory Documentation should include:
A procedure for the investigation of Out of Specification and Out Of Trend results;
6.9 Some kinds of data (e.g. tests results, yields, environmental controls) should be recorded in
a manner permitting trend evaluation. Any out of trend or out of specification data should be
addressed and subject to investigation.
16
Guidance
MHRA OOS Guidance
FDA
DI Guidance
MHRA GMP
WHO
FDA Q&A
MHRA GxP
PIC/s
EMA Q&A
January 2015
September 2015
April 2016
July, 2016
August 10, 2016
August 11, 2016
ALCOA+
Accurate
Legible
Contemporaneous (real time)
Original
Attributable
19
✓Accurate
✓Complete
✓Consistent
✓Secure
STRATEGY – DEFINE, EDUCATE, COMMUNICATE
Controversial TopicsBE TRANSPARENT: NEVER delete, alter or hide data
Invalidation of results
because of laboratory error: under what conditions
Reintegration
Remeasurement, reinjection
Repreparation
Retest
Resample
Reportable result
20
Controversial Topics
Use of averaging?
Definition of reportable values?
Number of retests?
Second analyst?
Use of outlier testing?
Defining testing into compliance?
21
FDA Guidance
22
The Barr Court Case
From the New York Times
February 6, 1993, Saturday
(AP); Financial Desk
COMPANY NEWS; Judge Rules On Barr Labs
A generic drug manufacturer must recall batches of some of its medicines and stop distributingothers until the company completes studies of itsmanufacturing process, a Federal judge ruled onThursday. But United States District Judge Alfred M. Wolin refused a request by Federal pharmaceuticalregulators to order a complete shutdown
23
Barr and OOS
Faced with potential closure, the company
took FDA to court
The judge went into great details as to the
meaning and implications of OOS results
The outcome: FDA draft guidance: 1998
FDA final guidance: 2006
24
Why the Guide?
FDA’s current thinking on how to evaluate out-of-specification (OOS) test results
The term OOS results includes all test results that fall outside the specifications or
acceptance criteria established in drug applications, drug master files (DMFs), official
compendia, or by the manufacturer
The term also applies to all in-process laboratory tests that are outside of established
specifications
25
Applicability of Guide
Applies to chemistry-based laboratory testing of drugs regulated by CDER
Directed toward traditional drug testing and release methods
Can we use it for:
Microbiological
Biological
Physical
Qualitative
tests?
26
The Guide
Laboratory tests are performed on active pharmaceutical ingredients, excipients and
other components, in-process materials, and finished drug products to the extent that
current good manufacturing practice (CGMP) regulations (21 CFR parts 210 and 211) and
the Federal Food, Drug, and Cosmetic Act (the Act) (section 501(a)(2)(B)) apply
Laboratory testing, required by CGMP regulations (211.160 and 211.165) is necessary to
confirm that components, containers and closures, in-process materials and finished
products conform to specifications, including stability specifications
Testing also supports analytical and process validation efforts
27
Why does your company perform Tests?
What is the purpose of testing?
Can you assure a correct result?
What is a correct result?
28
Why Does your company Perform Tests?
To determine if a critical quality attribute conforms or does not conform with its
specification
You cannot assure (at a 100% confidence level) that you will obtain a correct result
The correct result is that which is identical to the true result
but
You never know the true result because if you did… you wouldn’t need to test
29
Uncertainty
Since you cannot assure a correct result there is always some level of UNCERTAINTY
associated with any reported laboratory test result
Uncertainty represents risk
What is the risk associated with reporting an incorrect test result?
30
The risk
Declare that a result conforms with the specification when the TRUE result actually does
NOT conform
Declare that a result does not conform with the specification when the TRUE result actually
DOES conform
(False positive / False negative)
31
Some Definitions
ReMeasurement:
ReTest
ReSample
ABORTED TEST
INVALID TEST
INVALIDATED
TEST RESULT
Reinject same sample preparation
Additional test(s) on additional
aliquot of
same original sample
New sample
You stop the test e.g. spillage
Test acceptance criteria fail
(tells you nothing about the results)
Found conclusive evidence of
laboratory error: method, reagents,
performance of test, equipment etc.32/
Reportable Result – The Guide
“The term reportable result as used in this document means a final analytical result
This result is appropriately defined in the written approved test method and derived from
one full execution of that method, starting from the original sample”
33
USP Definition of a Reportable Value
A reportable value is often a summary value for several individual determinations
It is the end result of a completed measurement method, as documented
It is the value compared with the acceptance criteria
In most cases, the reportable value is used as documentation for internal and external
users
34
What is an OOS Results
A reportable result (obtained after applying the entire method as described and
performing any calculations and rounding off the significant figures) which when
compared to the specification is outside the limit / range or declared value
35
Ever heard this….
“The Lab don’t know how to test”
“ Give it to Sheila…
he knows how to do
that test…
Bruce always gets bad
results!”
36
What Could be the Cause of An OOS
1. _____________________________
2. _____________________________
3. _____________________________
Which is most likely?
37
How is Testing Performed
Collect a sample from the batch
Take part of sample (an aliquot) and test e.g. for a liquid: measure out a portion; inject
into HPLC
or
For a powder, take part of the sample (weigh it); dissolve in diluent and inject to HPLC
or
For a tablet: crush the tablet to make a powder; add diluent to dissolve the powder; inject
38
How is Testing Performed
Can take one aliquot and prepare for testing
Can take duplicate aliquots (two weighings) and prepare for testing
Can make duplicate injections (two injections)
39
How is Testing Performed
In which case, the method will describe the reportable result as…
Either the average result of the two weighings and two injections with a standard
deviation
or
The average result of each individual preparation (i.e. each preparation / weighing) and
not more than a certain RSD between the two (or more) results
40
Simplest Scenario (1):
One measurement, one preparation , one injection
41
Scenario (2):
3 preps; 3 injections, ONE reportable result
42
Scenario (3):
3 preps, 3 injections; one reinjection of preparation;
repreparation from weighed / crushed powder?
43
Apparent Implications
The individual determinations do not have to meet the criteria of the reportable result
Determinations are not generally reported out of the QC laboratory
The variability of determinations is similar to a system suitability issue
Set a limit on the standard deviation
44
Apparent Implications
All reportable results must be documented
Do not average OOS with in spec to get an in spec results to release the batch
Do not average reportable results for QA to make a batch release decision
QA must see all reportable results obtained during testing
If after QA makes a decision, and a value is needed for a COA, then average them
45
OOS Guide
The need to provide all reportable results to the QCU is
reinforced
“In addition, when investigation by a contract laboratory
does not determine an assignable cause, all test results
should be reported to the customer on the certificate of
analysis”
46
Before Barr – Current Practice
Prior to 1993 and the court decision – it was
COMMON practice to retest once
or
in exceptionally good companies twice
and to release the batch if the retest result was
within the specification
Companies had not really thought about the
practice
But then…nor had the regulators
47
Barr: What happened in court
The judge heard experts on
behalf of FDA and Barr regarding
the practice of retesting
FDA wanted retesting to be banned
under all circumstances
After a long hearing at which five industry experts, an
FDA investigator, and several company employees
testified, Judge Alfred M. Wolin, U.S. District Judge for
the District of New Jersey, issued a 79-page opinion 48
The Barr Court Case 1993 – Data Integrity?
Reported problems include
misplaced records
test data recorded on scrap paper
failure to control manufacturing steps such as
those governing products' physical properties
release of products not meeting their
specifications
inadequate investigation of failed products
49
Barr: “Testing into Compliance”
Barr had numerous failures
Performed retests with
no investigations
no regard for process and product history
Tested until results met specifications
Then irrespective of previous OOS results
for the batch, released product
reporting only the passing results
50
Q: How do you report passing OOS’s on COA?
Reading the Judgment
The Barr court judgment is like reading FDA’s guidance
Judge Wolin preferred "out-of-specification" (OOS) laboratory results rather than "product failure" used by FDA's investigators
Ruled an OOS result identified as laboratory error by investigation or an outlier test, or overcome by retesting is not a product failureBUT
Limited situations where laboratory error could be used 51
Guide to Inspection: QC Labs
Issued July 1993 (must have been working on
it while the court case was ongoing)
Addresses OOS results; tells inspectors to be alert
“Evaluate the company's system to investigate laboratory test
failures
These investigations represent a key issue in deciding whether a
product may be released or rejected and form the basis for retesting, and resampling “
52
Guide to Inspection: QC Labs
OOS results fall into three categories:
laboratory error
non-process related or operator error
process related or manufacturing process error
Evaluate the company's retesting SOP for compliance with scientifically sound and appropriate procedures
53
Guide to Inspection: QC Labs
A very important ruling in one recent court decision sets forth a procedure to govern the retesting program
This district court ruling provides an excellent guide to use in evaluating some aspects of a pharmaceutical laboratory, but should not be considered as law, regulation or binding legal precedent
The court ruled that a firm should have predetermined testing procedure and should consider a point where testing ends and product is evaluated. If results are not satisfactory, product is rejected
54
Collect Data – Be Objective
It is a capital mistake to theorize before one has data. Insensibly one begins to
twist facts to suit theories, instead of theories to suit facts.
Sir Arthur Conan Doyle (1859 – 1930)
Author of the Sherlock Holmes detective series
55
The FDA Guidance
III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY
INVESTIGATION
The source of the OOS result should be identified either as an aberration of
the measurement process or an aberration of the manufacturing process
Even if a batch is rejected based on an OOS result, the investigation is
necessary to determine if the result is associated with other batches of the
same drug product or other products
Batch rejection does not negate the need to perform the investigation
The regulations require that a written record of the investigation be made,
including the conclusions and follow-up (§ 211.192)
56
Investigation Retesting
Hypothesis Testing
If you have a hypothesis, you can test it
That is NOT what is generally called retesting and usually should not be performed to
obtain a passing result but to recreate the FAILING result
The FDA Guidance
III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY
INVESTIGATION
To be meaningful, the investigation should be thorough, timely, unbiased,
well-documented, and scientifically sound
The first phase of such an investigation should include an initial assessment
of the accuracy of the laboratory's data
Whenever possible, this should be done before test preparations (including
the composite or the homogenous source of the aliquot tested) are
discarded.
This way, hypotheses regarding laboratory error or instrument malfunctions
can be tested using the same test preparations.
58
Initial Assessment of OOS Result
Bear in mind prior:
Product history
Process history
Test history
Reliability of equipment
Reliability of the analyst
Precision of the test (validation)
59
The Guide
The first responsibility for achieving accurate laboratory testing results lies with the analyst
who is performing the test
The analyst should be aware of potential problems that could occur during the testing
process and should watch for problems that could create inaccurate results
In accordance with the CGMP regulations in § 211.160 (b)(4), the analyst should ensure
that only those instruments meeting established performance specifications are used and
that all instruments are properly calibrated
60
Responsibility of Analyst
To follow test procedure as written
To be alert to errors and STOP test BEFORE obtaining the result if error is suspected,
recording what happened e.g. spill
Analyst responsible for ensuring that
instruments meet performance specifications
and are properly calibrated [KSG: maintained?]
Once an OOS result is obtained to review all
records relative to the test to identify possible
laboratory error
61
Supervisory Role: USE a checklist
Visit the crime scene
Supervisors / team leaders / laboratory head:
Should be experienced analysts
Frequently audit while tests ARE BEING
performed
in order to be able to objectively investigate
OOS results
62
Responsibility of Supervisor
Objective assessment without
preconceived assumptions as to cause of OOS
Immediate assessment may include:
Re-examination of:
actual solutions
Test units
Glassware
used in the original measurements and preparations
This could provide more credibility for laboratory error hypotheses
63
Supervisor (The Guide)
1. Discuss the test method with the analyst; confirm analyst knowledge of and performance of the correct procedure.
2. Examine the raw data obtained in the analysis, including chromatograms and spectra, and identify anomalous or suspect information.
3. Verify that the calculations used to convert raw data values into a final test result are scientifically sound, appropriate, and correct; also determine if unauthorized or unvalidatedchanges have been made to automated calculation methods.
4. Confirm the performance of the instruments.
5. Determine that appropriate reference standards, solvents, reagents, and other solutions were used and that they met quality control specifications.
6. Evaluate the performance of the test method to ensure that it is performing according to the standard expected based on method validation data and historical data.
7. Fully document and preserve records of this laboratory assessment.
64
Responsibility of Supervisor
To INVESTIGATE:
Review notebook / worksheet with analyst :
Was method was followed: with a copy of themethod in your hand, have the analyst describeexactly how they performed each step:confirm that the method was understood & followed
Review raw data: Perform calculations againincluding checking dilution schemes
Unauthorised changes to automated calculations
Examine reagents, (reference) standards, solutions
Examine glassware
Performance of instruments
65
The FDA Guidance
III. IDENTIFYING AND ASSESSING OOS TEST RESULTS — PHASE I: LABORATORY
INVESTIGATION
If this initial assessment indicates that no meaningful errors were made in
the analytical method used to arrive at the data, a full-scale OOS
investigation should be conducted
66
Invalidate result
Perform new test on same sample
Convincing evidence
of
Laboratory Error
Correct if possible
or
Reject Batch
Production Error
Resample
Double sample
Revise sampling procedures
Sampling Error
QA decision
regarding
batch disposition
All within
specifications
Reject Batch
One result
OOS
Retest
??? further aliquots
from original sample
Inconclusive
No evidence of production error
QA Investigation
Inconclusive
Laboratory Investigation
Report to QA (copy in batch file?)
OOS resultOOS Flow Chart
67
Out of Specification Results
If the laboratory investigation is conclusive
Document findings
INVALIDATE original test
Perform NEW test on same sample
Report original result with investigation as well as new result in batch record for QA review prior to release
COA carries new result only
If the laboratory investigation is NOT conclusiveinform QA (or customer for contract lab) 68
Out of Trend Results
If the laboratory investigation is conclusive orinconclusive, consult with QA
In most cases, DO NOT perform any additionaltesting or sampling
Make product disposition judgment based on:
Original result
Product history (e.g. stability data – statistical analyses of particular use here)
Batch history (e.g. review indicates that there were noprocessing errors / there were errors)
Other investigational findings
69
FDA: Full-Scale OOS
Investigation
Use a pre-defined procedure
Production/ process review and / or additionallaboratory work
Identify root cause and implement CAPA
QA / QCU responsibility, includes CMOs if used
Documented in the batch record
Involves all aspects of manufacture, qualitycontrol and sampling
Describes corrective actions and endpoint
Is performed PRIOR to ANY retesting
70
FDA: Full-Scale OOS
Investigation
If cause of OOS is identified, batch is rejected
In this case need CAPA on process / product
May not identify cause and may need additionallab testing:
Retest additional portion of original sample
Resample
71
FDA: Retesting and Resampling
Use another analyst? Where possible
The maximum number of retests should be
specified in advance in an SOP
May, on rare occasions, deviate from SOP but
with documented rationale and protocol
The number may vary depending upon the
variability of the test method and NOT
depending on the results obtained
Resampling raises questions as to sampling
procedure validity
72
OOS Guidance
Is there a fundamental process flaw?
Product or process redesign is addressed with a new paragraph.
“OOS results may indicate a flaw in product or process design. … In such cases, it is
essential that redesign of the product or process be undertaken to ensure reproducible
product quality.”
73
A horrible question – but needs to be
asked…AND answered…honestly
Retesting
Retesting has an additional requirement.
“The maximum number of retests to be performed on a sample should be specified in advance in a written standard operating procedure (SOP)
Any deviation from this SOP should be rare
In such cases, before starting additional retesting, a protocol should be prepared that describes the additional testing to be performed and specifies the scientific and/or technical handling of the data”
74
Retesting – How many repeats?
While the guidance does not give
recommendations for the sample size for
retesting, the example given uses seven
retests
Seven was the suggestion in a footnote in
the Barr Case judgment
The sample size question is still unresolved
75
Resampling
“The original sample from a batch should be sufficiently large to
accommodate additional testing in the event an OOS result is
obtained
In some situations, however, it may be appropriate to collect a new
sample from the batch.”
76
Be very cautious about collecting a new sample
How to Identify OOT Results
Out of Trend or unusual results are generallyresults that:
MEET the product specification(may be borderline)
ARE outside the control limits of the process,where control charts are used – iMOH requiring this routinely…or
ARE different to results usually obtainede.g. spec: 95.0 – 110.0usual results: 98.5 – 101.0OOT result: 96.4
77
FDA: Concluding the Investigation
Evaluate results and determine batch quality
Release decision by QCU / QA
An initial OOS does not necessarily mean
that the batch fails and must be rejected.
Where the suspect result is invalidated, the
result should not be used to evaluate the
quality of the batch or lot
For inconclusive results – give full
consideration to the OOS result
78/87
FDA: Concluding the Investigation
Example given shows seven retest results
which gives the only indication in the guide
regarding numbers of retests
The example given is also extreme:
89.5% OOS
99.0, 98.9, 99.0, 99.1, 98.8, 99.1, 99.0%
Consider method precision and validation data
in making release / reject decision
79/87
FDA: Cautions
Where a series of assay results (to produce
a single reportable result) have some
individual results OOS and some in spec and
all within the known method variability,
passing results no more likely to represent
the true value than the OOS result.
In this case the company must err on the side of caution and reject the batch
A result that is low but in specification should
also be a cause of concern
80/87
Cautions
The first paragraph continues a discussion of reportable results stating that “… a firm should
err on the side of caution …”
a low assay result should raise concern and “One cause of the result could be that the batch was not formulated properly. Batches must be formulated with the intent to provide not less than 100 percent of the labeled or established amount of active ingredient”
81
Take Away Information
Companies must have an up-to-date SOP
Write the SOP at the 8th grade level
Use simple declarative sentences.
Use the wording directly out of the guidance where possible; copyright free!
Require the operational definition of the reportable result to be in the SOP, analytical test method, retest protocol and reports
82
Quality Metrics: are you ready?
report number of invalidations for lab error
Revisit your investigations
With an open mind
Ask:
Why did we get an OOS result in the first place
Could it have been laboratory error? YES
Could it have been production error? YES
Did we ELIMINATE the possibility of production error?
Did we CONCLUSIVELY demonstrate lab error?
83
Thank you for your Attention
Questions?
84/
OOS Keywords
Data governance
Data integrity
ALCOA +
OOS
OOT
Questionable / unusual / atypical result
Reportable result
Retest
New test
Invalid test
Aborted test
New test
Resample
Batch release
Investigation
Laboratory error
Production error
Sampling error
Data integrity
Data quality
Averaging
Training, qualification, education
COA
Reporting Results
Related Documents
![[Zelda.com.Br] OoT - Taiwan](https://static.cupdf.com/doc/110x72/55cf9027550346703ba36182/zeldacombr-oot-taiwan.jpg)
