Thoughts on the Use of Decision Analysis in the Review of New Drug Applications October 3, 2007 Todd Durham.

Thoughts on the Use of Decision Analysis in the Review of New Drug Applications

October 3, 2007

Todd Durham

Outline

NDAs and the nature of the decisionPotential benefits and challenges of

decision analysis An illustrationLearning and opportunities

Mental Exercise

Imagine that tomorrow you are diagnosed with a disease from which you will die in exactly 7 days.

If you could take a pill: That would definitely cure you from this disease,

how much would you pay for it? That would give you a 25% chance of a cure, how

much would you pay for it? That would give you a 25% chance of a cure, how

much risk (s%) of a debilitating stroke would you accept?

New Drug Applications

Marketing applications for new drugs FDA reviewed between 20-30 NDAs (for

NMEs) per year between 2001-2003 (FDA, Critical Path, 2004)

Data submitted with a NDA Human evidence of benefit Human evidence of risk Manufacturing controls Animal data on toxicology and carcinogenicity

Objective in Reviewing a NDA

Decide if a drug’s benefits outweigh its risks

Evolved historically with various changes in the law to:Avoid misleading doctors or consumersKeep dangerous drugs out of the system

What does the law really say?

“Substantial Evidence” from FDC Act of 1962

Substantial evidence was defined in section 505(d) of the Act as “evidence consisting of adequate and well-controlled investigations, including clinical investigations, by experts qualified by scientific training and experience to evaluate the effectiveness of the drug involved, on the basis of which it could fairly and responsibly be concluded by such experts that the drug will have the effect it purports or is represented to have under the conditions of use prescribed, recommended, or suggested in the labeling or proposed labeling thereof.”

(FDA, Clinical Evidence of Effectiveness, 1998)

Sufficient Criteria for Demonstration of Efficacy

Choice of Primary Endpoint Reliably measures a clinically relevant characteristic Statistically sensitive to treatment Identified a priori (with corresponding analysis methods)

Results for Primary Endpoint Treatment effect is “statistically significant” in at least two studies Magnitude of treatment effect (Δ) is clinically relevant

Results for Secondary Endpoints Results from secondary endpoints further describe the relevance

of Δ (primary endpoint) if results from primary endpoint in the same study are statistically significant

The Case of Carvedilol

“… the usual two-study FDA paradigm does not make sense under all situations. This much is clear. But I would also suggest, as stated above, that experience has shown the paradigm to be a very useful guideline; exceptions should therefore be relatively unusual, and, when in doubt; one should err on the side of conservatism. Nevertheless, it strikes me as absurd in extreme cases to insist that if one does not meet the original primary end point in a study, that conclusions can never be definitive but only hypothesis generating.” (Fisher, 1999)

Criteria Used in Reviewing a NDA

Benefit Quantity of evidence Quality of evidence Typically restricted to one or a few “endpoints” Leads to a labeled claim consistent with results

Safety From any number of reported adverse events Cardiac safety studies (e.g., QTc) Potentially animal studies (e.g., risk to fetus)

Manufacturing

Decision to be Made

Approve the new drugReject the new drugAsk the sponsor for more information

(“approvable”)

Influences on the Decision

Statistical robustness of the apparent benefit, with appropriate statistical control of the false positive rate

Clinical relevance of the benefit Excess safety risks, with no control of the false

positive rate Severity of the disease Availability of other treatments

When a Drug is Approved

Can be legally marketed in the U.S. Doctors have a prescribing option Patients have a treatment option Pharmaceutical companies make revenue Need for education all around

Safety will continue to be monitored Surveillance has less rigor than RCTs

May be studied further Expand the label Clarify the role of the new drug or its effects

When a Drug is “Approvable”

Can not be legally marketed in the U.S.Doctors can not prescribe itPatients can not take it

May be studied furtherPharmaceutical companies spend more

money on researchTime for further research and submission

When a Drug is Rejected

Sponsor may withdraw applicationCan not be legally marketed in the U.S.

Doctors can not prescribe itPatients can not take it

“Easy” Approval Decisions

A lot of evidence of clear benefit Clinically relevant Statistically robust (very unlikely due to chance)

At least moderately sized safety database reflects reasonable risks

No evidence of toxic or carcinogenic effects No other available treatments or just a few

treatments with some toxicities

“Easy” Rejection Decisions

Obvious hazards with little benefitPoor manufacturing controls

Decisions are Much Harder When

Mixed results for benefitDrug which has been shown to have a

benefit in some populations but not others.A lot of studies, only a few of which were

successful.Statistical criteria for “success” are not met.

Some significant trade-offs must be reckoned with.

Made Even More Difficult

Changing landscape Regulatory standards (e.g., emerging concerns) Medical advances Changing standard of care Ex-US medical care

External pressures Congress Patient advocates Pharmaceutical industry

Benefits of a Decision Analysis

Transparency of the decision Many objectives possible (identified, weighting) Influences for all stakeholders

Role of uncertainties Which ones make the most difference?

Model that can be applied to many products in the same therapeutic area, but evolve over time.

Dissection of the problem greater understanding

Transparency

PatientsTo pharmaceutical companyWithin the FDACongress

Role of Uncertainties

How much do the following uncertainties bear on the consequences? Quality or quantity of evidence of benefit Medical need, population affected Available therapies How many patients will actually use the treatment?

Don’t need to be accurate but having a grasp on the range of uncertainties can still be instructive (through tornado diagrams)

An Evolving Model

Changes in medicineChanges in how medical expenses are

reimbursedChanges in societal priorities or norms

Dissection of the Problem

Factors which most influence the best decision can lead to new prioritiesRole of available therapy compare the

new treatment to available therapyQuantity of evidence additional

informationThe safety/benefit tradeoff patient

involvementInsensitivity of the model to various

uncertainties can make decisions easier

Challenges of DA for this Application

How to define the safety risks?All of them?Control of false positive rate?

How to assess the consequencesBy whom?Using what measure?

Consequences

TimeMoneyHuman livesUnwanted eventsQuality-adjusted life yearsCredibility / trust (how to value?)Quality of information (what is its value?)

Basic Decision Tree

NDA Decision

Approved

Approvable

Rejected

Consequences

C1 C2 C3

Waiting for More Information

NDA Decision

Approved

Approvable

Rejected

New Study?

OutcomeYes

No

Success

Failure

Presumably, success would lead to a greater chance of regulatory approval, but what are the consequences of having made this decision to wait for more information?

Illustration: Serious Diagnosis

Advanced cancer that affects 50,000 individuals per year

Current expected life-span (median) is 20 months from diagnosis.

The one available treatment is not tolerated well such that most patients choose not to take it.

Loosely adapted from story in New York Times, 2007.

Results from Clinical Trials

New treatment compared to placebo Efficacy:

Treatment effect is ~4 months of survival (benefit) in two studies.

In one study survival had a nominal p-value <=0.050, but it was a secondary endpoint.

Primary endpoint was stopping progression of cancer (failed in both studies).

Safety: Most common side effect is flu-like symptoms 1-2% chance of a stroke from new treatment

Considerations

DA could address the consequences of a world with (now or later) and without the new treatment Lives lost in a period of time New strokes in a period of time Bouts of flu-like symptoms

Was survival a false positive finding? Zero survival benefit What to do with the conventional hypothesis testing

interpretation? Won’t the benefit depend on how many patients might

use the new treatment?

Could this Ever Be Applied?

Modest proposals: FDA could conduct an exercise by writing out an influence

diagram for approval decisions in one therapeutic area. Carry out research on how to best communicate risk to

patients (both benefit and safety). Increased emphasis on risk communication to patients.

Steiner, 1999 has tremendous insight on the topic. More difficult proposal:

Conduct focus groups with patients to examine ability to elicit their trade-offs. Howard has written on ways to value life and other outcomes.

Fantasy-land proposal: Make all drugs available for marketing and change the

regulatory paradigm such that regulators verify accuracy of labeling and educate doctors and the public.

Learning from Experience

Unexpected clarity, almost profound new understanding of the decision to be made.

Ability to proceed without regret knowing the problem had been understood as best as humanly possible.

Training is important. Even highly intelligent people do a poor job of estimating uncertain quantities.

Illustration: What if…

The benefit was only 0-4 months of survival, with a great deal of skepticism that 4 months from the trials was “real”?

Some patients might trade the chance of a stroke for a chance at an extra month or two of life.But they can’t make this choice unless the

drug is made available to them.We won’t know unless we ask.

References

US Department of Health and Human Services, Food and Drug Administration, 2004. Challenge and opportunity on the critical path to new medical products. Available from www.fda.gov.

US Department of Health and Human Services, Food and Drug Administration, 1998, Providing clinical evidence of effectiveness for human drug and biological products. Available from www.fda.gov.

Fisher L. Carvedilol and the Food and Drug Administration (FDA) Approval Process: The FDA Paradigm and Reflections on Hypothesis Testing. Controlled Clinical Trials 1999;20:16–39.

Steiner J. Talking About Treatment: The Language of Populations and the Language of Individuals. Annals of Internal Medicine 1999; 130,7: 618-622.

Howard RA. On Making Life and Death Decisions. Readings on the Principles and Applications of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic Decisions Group.

Howard RA. On Fates Comparable to Death. Readings on the Principles and Applications of Decision Analysis. Howard RA and Matheson JA, editors. 1989. Strategic Decisions Group.

Andrew Pollack, “Panel Endorses New Anti-Tumor Treatment,” The New York Times (March 30, 2007).