Data Analysis Of An Analytical Method Transfer To

Data Analysis of an Analytical Method Transfer to Two Labs

Comparability by testing absolute tolerance limits, equivalence of

means, or a combination of both.

Purpose of Presentation

To provide the basis for a PDA task force discussion to arrive at a

consensus of best industry practices for data analysis of method transfers.

The discussion is also relevant to method validation activities.

Introduction:Mitigating Risks to Technology Transfer

• Comparisons of methods is discussed in USP <1010>, but comparison of laboratories is not.

• In USP and ICH, accuracy is defined only in context to unbiasedness (trueness). In ISO, accuracy combines the concept of unbiasedness and precision, a form of total error.

• Comparison of laboratories should be considered in regards to the phase of development of the drug product.

Introduction:Mitigating Risks to Technology Transfer

• Acceptable criteria should reflect the intended purpose of the method and control the risk of incorrectly accepting an unsuitable analytical method.

• For example, the bias of a stability indicating method, especially if the bias may indicate a different product expiry, should be considered as a higher risk.

Introduction:Mitigating Risks to Technology

Transfer: Phased Approach• Phased approach includes considerations of risk for

the different stages of development:– Pre-clinical through early Phase II

• Evolution of process development• Evolution of analytical methods

– Late Phase II through Phase IV

Note: Risks in context to the phased approach varies owing to drug product, within companies, and changes to guidance.

Introduction:Less Risk to Transfer Studies:

• When trending is unimportant.• When equivalent results is not necessary.• When deliberate changes are made to

processes and analytical methods within early development stages, Phase I and early Phase II.

Data Analysis

• The presentation reviews data analysis by use of:– Absolute tolerance limits that emphasize means

and intermediate precision within qualification acceptance criteria.

– The equivalence of means by the two one-sided t-test (TOST) is demonstrated.

– The use of both is considered within the context of the phased approach.

Introduction:Transfer Study Design

• The analytical method was transferred to two laboratories.

• The originating lab and receiving labs, n=3 labs, each tested n=6 homogenous samples (reported results were the mean of two replicates), split between n=2 analysts.

Introduction:Transfer Study Design

• For comparing the results within this presentation, actual acceptance criteria are not used; instead the data analysis assesses the acceptance criteria that the data supports.

• An assumption of the study is that the true value of the homogenous test sample is 4.0 mg/mL.

Raw Data

Accuracy and Precision Analysis

Sample Suitability Based on Replicate Precision

Intra-analyst (Inter-reportable result)

Intra-laboratory (Inter-analysts)

Inter-laboratory Precision

“Total Error” Analysis

Outlier Analysis

Equivalence of Means: TOST• Requires predetermination of an acceptance interval for the

lower and upper limits, this is called a practical difference threshold.

• The confidence intervals for the ratio of test/reference are often set based on convention (e.g., alpha = 0.10).

• Tests whether the measured bias is within the acceptance interval.

• Compared to the t-test, the null and alternative hypothesis are reversed.– The type I error is the probability of erroneous acceptance of

equivalence.– Type II error is the probability of erroneous acceptance of

nonequivalence.

• A matched pairs study design is required for TOST, analysis by absolute tolerance limits does does not. Some forms of TOST allow for asymmetric study design

• TOST is discussed in USP <1010> but it is not appointed as to its appropriateness for evaluating equivalence for laboratories or methods. It is discussed in ICH in regards to bioequivalence testing.

• In USP and ICH, accuracy is defined only in context to unbiasedness (trueness). In ISO, accuracy combines the concept of unbiasedness and precision.

Equivalence of Means: TOST

Assumption for Superiority of Equivalence of Means Testing

• “Reduces risk of wrongly concluding equivalence when in fact two laboratories or two methods are not equivalent.”

Feng, Liang, Kinser, Newland, and Guilbaud. Anal Bioanal Chem (2006) 385:975-981.

TOST Analysis Using JMP

• JMP describes the TOST test as “Practical Difference” testing and warns that confirming no difference in means is impossible.

• This form of TOST demands that an interval around the hypothesized value is chosen.

• The test tries to show that the mean is not outside the interval.

• The key to success is that the desired control around the mean is successfully selected.

• JMP equivalence of means testing does not require a symmetric study design.

TOST Analysis Using JMPTo perform a TOST test:

1)Do a one-sided t-test that the mean is the low value of the interval, with an upper tail alternative.

2)Do a one-sided t-test that the mean is the high value of the interval, with a lower tail alternative.

3) If both tests are significant at some level α, then you can conclude that the mean is out-side the interval with probability less than or equal to α, the significant level. In other words, the mean is not significantly practically different from the hypothesized value, or, in still other words, the mean is practically equivalent to the hypothesized value. (Technically, the test works by a union intersection rule, whose description is beyond the scope of this book.)

--JMP Start Statistics: A Guide to Statistics and Data Analysis Using JMP by John Sall, Lee Creighton,

Ann Lehman

Jmp: Oneway Analysis of Results

Jmp : Oneway Analysis of Results

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4.The alpha level was set at 0.10.

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4.The alpha level was set at 0.10.

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.4.The alpha level was set at 0.10.

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8.The alpha level was set at 0.20.

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8.The alpha level was set at 0.20.

Jmp: Practical Equivalence

Difference considered practically zero, “Specified Practical Difference Threshold)” = 0.8.The alpha level was set at 0.20.

Practical Equivalence

• The equivalence of means testing provides different conclusions based on:– The specified practical difference threshold (e.g., ±5,

10, 15, or 20% (converted back to the units of the test result), and

– The alpha level (e.g., 0.05, 0.10, 0.15, or 0.20).• At 10%, labs 1 and 3 are equivalent to lab 2, but

not each other.• If lab 2 was the originating lab, this problem could

be a challenge, albeit an interesting one.

Summary

• Both methods of data analysis must meet the challenge of setting appropriate acceptance criteria based on risk to include the phase of development of the drug product.

• Both methods of data analysis indicate that the labs do not meet an acceptance criteria for accuracy of 90 to 110% nominal, but do for 80 to 120% nominal.1

Summary: Absolute Limits

• Setting acceptance criteria may not be easy, but it is easier using absolute limits because the units are the same as the test results.

• Absolute limits analysis provides estimates of precision for the critical components of error.– Useful to optimize training, sample handling, or the method.

• Means of sub-groups and the grand mean illustrate the bias between the labs, but within the acceptance criteria interval. It does not provide a confidence interval.1

Summary: Absolute Limits

• For early phase technology transfer, acceptance criteria using absolute limits has many advantages including:– Intuitive– Ease of use– Provides a more detailed analysis of sources of

error.– Easily reviewed by QA

Summary: TOST

• Setting acceptance criteria is even harder, because the units are not the same as the test results nor is it intuitive where to set the additional input of alpha.

• Does not provide estimates of precision for the critical components of error other than inter-laboratory.– Data analysis does not add value in regards to

investigations to improve assay performance.

• Bias between the labs must meet a practical difference threshold as well as a confidence level.1

Summary: TOST

• Complicated statistics do not drive quality. TOST is often intimidating, not assessable to some companies, misused, and therefore may be misleading.

• Later phase technology transfer may benefit from acceptance criteria using absolute limits plus the additional TOST inputs and TOST data analysis because it adds an estimate of confidence intervals. Equivalence testing should rarely be used as the basis of acceptance criteria.

Optional Slides

• Slides that follow are for further discussions that are of interest to the task force for the PDA AMD TR.

Total Error and Phased Approach

• Total error approach should be considered to include intermediate precision , bias within laboratories, and equivalency of means.

• A phased approach for data analysis may consider the use of equivalency testing and statistical determination of the robustness of study design. Studies should provide greater probability of correctly concluding that two identical methods are equivalent for Phase III and even late Phase II development stages.

Equivalence of Means Analysis

• The Schuirmann’s TOST is the standard approach for bioequivalence testing.

• The acceptance criteria for bioequivalence testing may be consistent with comparability and tech transfer studies of potency/content and stability indicating assays.

• Equivalence testing is often considered as a total error approach.

Equivalence of Means Analysis

• Hoffman and Kringle (H&K) argue that “typical acceptance criteria for analytical method precision and accuracy are not chosen with regard to the concept of method suitability and are commonly based on ad-hoc rules” which they say is inadequate.

• H&K: “Although such ad-hoc approaches may meet regulatory requirements, they yield unknown and uncontrolled risks of rejecting suitable bioanalytical methods (producer risk) and accepting unsuitable bioanalytical methods (consumer risk). “

• H&K: “Current criteria are based on observed estimates of bias and variability, rather than on the true method bias and variability.”

Introduction:Mitigating Risks to Technology Transfer

• Random and systematic variability should be addressed in context of the measurement process and the analyte measured.

• Statistical measures should address the direction and magnitude of the errors to include the mean and the standard deviation, or the expressions derived from them (e.g., %RSD).

• Estimated variability can be used to calculate confidence intervals for the mean and tolerance intervals to capture an estimate of the specified proportion of the individual measurements.