OVERVIEW

STATISTICAL CONSIDERATIONS IN POSTMARKETING SAFETY EVALUATION

A. Lawrence GouldMerck Research Laboratories

West Point, PA [[email protected]]

FDA/Industry Workshop29 September 2006

Washington, DC

September 29, 2006

2

OVERVIEW

September 29, 2006

3

Spontaneous AE Reports

• Clinical trial safety information is limited & relatively short duration

• Safety data collection continues after drug approval o Detect rare adverse eventso Obtain tolerability information in a broader

population

• Large amount of low-quality data collected

o Not usable for trt comparisons or risk assessment

o Unknown sensitivity & specificity

• Evaluation by skilled clinicians & epidemiologists

• Long history of research on issue

September 29, 2006

4

Information Available Postmarketing

• Previously undetected adverse and beneficial effects that may be uncommon or delayed, i.e., emerging only after extended treatment

• Patterns of drug utilization

• Effect of drug overdoses

• Clinical experience with study drugs in their “natural” environment

September 29, 2006

5

The Pharmacovigilance Process

Detect SignalsTraditional Methods

DataMining

Generate Hypotheses

Refute/Verify

Type A (Mechanism-based)

Type B(Idiosyncratic)

Insight from Outliers

EstimateIncidence

Public HealthImpact, Benefit/Risk

Act

Inform

Change LabelRestrict use/

withdraw

September 29, 2006

6

Considerations & Issues (An Incomplete List!)

• Incomplete reports of events, not reactions

• Bias & noise in system

• Difficult to estimate incidence because no. of pats at risk, pat-yrs of exposure seldom reliable

• Significant under reporting (esp. OTC)

• Synonyms for drugs & events → sensitivity loss

• Duplicate reporting

• No certainty that a drug caused the reaction reported

• Cannot use accumulated reports to calculate incidence, estimate drug risk, or compare drugs

September 29, 2006

7

DATA MINING

September 29, 2006

8

Data Mining is a Part of Pharmacovigilance

• Identify subtle associations (e.g., drug+drug+event) and complex relationships not apparent by simple summary

• Identify potential toxicity early

• Finding ‘real’ D-E associations similar to finding potential active compounds or expressed genes – not exactly the same (no H0) – more like model selection

• Still need initial case review

respond to reports involving severe, potential life-threatening events eg., Stevens-Johnson syndrome, agranulocytosis, anaphylactic shock

• Clinical/biological/epidemiological verification of apparent associations is essential

September 29, 2006

9

Typical Data Display

No. Reports

Target AE

Other AE

Total

Target Drug

a b nTD

Other Drug

c d nOD

Total nTA nOA nSome possibilities

Reporting Ratio: E(a) = nTD nTA/nProportional Reporting Ratio: E(a) = nTD c/nODOdds Ratio: E(a) = b c/d• Need to accommodate uncertainty, especially if a is small• Bayesian approaches provide a way to do this

Basic idea:

Flag when

R = a/E(a) is “large”

September 29, 2006

10

Currently Used Bayesian Approaches

• Empirical Bayes (DuMouchel, 1998) & WHO (Bate, 1998)

• Both use ratio nij / Eij where

nij = no. of reports mentioning both drug i & event j

Eij = expected no. of reports of drug i & event j

• Both report features of posterior dist’n of ‘information criterion’

ICij = log2 nij / Eij = PRRij

• Eij usually computed assuming drug i & event j are mentioned independently

• Ratio > 1 (IC > 0) combination mentioned more often than expected if independent

September 29, 2006

11

Comparative Example (DuMouchel, 1998)

• No. Reports = 4,864,480, Mentioning drug = 85,304

Headache Polyneuritis

Reports AE Both AE Both

Mentioning 71,209 1,614 262 3

Reporting

Ratio

1.23 2.83

WHO FDA WHO FDA

Expected RR 1.29 1.23 0.76 1.42

5% Quantile -- 1.18 -- 0.58

Excess n 300 225 0 0

September 29, 2006

12

DATA MINING EXAMPLES INCORPORATING STATISTICAL

REFINEMENTS

September 29, 2006

13

Result From 6 Years of Reports on Lisinopril

Events w/Lower 5% RR Bnd > 2 (Bold N 100)

September 29, 2006

14

Persistence (& Reliability) of Early Signals

September 29, 2006

15

Accumulating Information over Time

0

0.5

1

1.5

2

2.5

3

3.5

4

Low

er 5%

RR

Bnd

dizziness

cough

palpitation

edema

angioedema

hyperkalemia

renal failure

incr. serum creatinine

• Lower 5% quantiles of RR stabilized fairly soon

September 29, 2006

16

Time-Sliced Evolution of Risk Ratios• See how values of criteria change over time within time

intervals of fixed length

Change in ICij for reports of selected events on A2A from 1995 to 2000tension = hypotension failure = heart failure

kalemia = hyperkalemiaedema = angioedema

0

0.5

1

1.5

2

2.5

3

3.5

4

Half-year interval

Exp

. RR

Cough

edema

kalemia

tension

Failure

September 29, 2006

17

Masking of AE-Drug Relationships (1)

• Company databases smaller than regulatory databases, more loaded with ‘similar’ drugs

eg, Drug A is 2nd generation version of Drug B, similar mechanism of action, many reports with B

• Elevated reporting frequency on Drug B could mask effect of Drug A

• May be useful to provide results when reports mentioning Drug B are omitted

September 29, 2006

18

Masking of AE-Drug Relationships (2)

September 29, 2006

19

Example 2: Vaccine-Vaccine Interaction

• From FDA VAERS database, reports from 1990-2002

• Intussusception is a serious intestinal malady observed to affect infants vaccinated against rotavirus

• Look at reports of intussusception that mention rotavirus vaccine (RV) and DTAP vaccine

• DTAP is a benign combination vaccine commonly administered to infants

• Demonstration question: Intussusception very commonly reported with RV – but does the reporting rate depend on whether DTAP was co-administered?

• Not easy to address using standard pharmacovigilance procedures

September 29, 2006

20

Outline of Analysis

• Standard tools provide intussusception reporting rate for pairs of vaccines, and for vaccines singly

• Result is a 3-way count table (corresponding to RV + or -, DTAP + or -, and intussusception + or -)

• Use log-linear model to see if intussusception is mentioned with the two vaccines together more often than the separate vaccine-intussusception reporting associations would predict

• Turns out that there is an association – Likelihood ratio chi-square is 17.41, 1 df, highly significant

September 29, 2006

21

Observed and Expected Report Rates

-2

0

2

4

6

8

Report R

ate

, %Observed Expected

RV+DTAP+

RV+DTAP-

RV -DTAP+

RV-DTAP-

Symbol area no. of reports

September 29, 2006

22

Comments

• Intussusception seems to be reported more often than expected when RV and DTAP are given together than when RV is given without DTAP, after adjusting for individual vaccine-intussusception associations

• Reports of intussception without RV are very rare, about 4.5/10,000 reports if RV is not mentioned

• The joint effect of RV and DTAP on intussusception reporting is small, but does reach statistical significance

• Not clear that apparent association means anything -- actual synergy between RV and DTAP seems unlikely, but explanation requires clinical knowledge

September 29, 2006

23

A NEW BAYESIAN APPROACH(Gould, Biometrical Journal 2006, to appear)

September 29, 2006

24

• ni = no. of reports mentioning i-th drug-event pair ~ Poisson (true for EB approach as well)

f(ni | Ei, i) = fPois(ni ; iEi)

• i drawn from a gamma(a0, b0) distribution or

from a gamma(a1, b1) distribution

o A model selection problemo Dist’ns reflect physician/epidemiologist’s

judgment as to what range of values corresponds to ‘signals’, and what does not

Model for Process Generating Observations

Expected count under

independence

Association measure

September 29, 2006

25

Prior/Model Density of • Bayes approach starts with a random mixture of

gamma densities,

f0( ; , a0, b0, a1, b1)

= (1 - )fgam(; a0, b0) + fgam(; a1, b1)

Use value of Ppost( = 1) for inference

• EB approach starts with expectation wrt given p nonrandom mixture of gamma densities,

f0( ; p, a0, b0, a1, b1)

= pfgam(; a0, b0) + (1-p)fgam(; a1, b1)

Use quantiles of posterior dist’n of for inference

Data determine parameter

values

Analyst specifies parameter

values

September 29, 2006

26

Comments • Bayes and EB approaches both model strength of

drug-event reporting assn as a gamma mixture

• Diagnostic properties of Bayes method can be determined analytically or by simulation

• Unknown separation of the true alternative dist’ns for more important than prior dist’n used for analysis

• Methods described here can be applied to other models – Scott & Berger (2005) used normal distributions – could also use binomial instead of Poisson, beta instead of gamma distributions to develop screening methods for AEs in clinical trials

September 29, 2006

27

DISCUSSION

September 29, 2006

28

Discussion

• Bayesian approaches may be useful for detecting possible emerging signals, especially with few events

• MCA (UK) currently uses PRR for monitoring emergence of drug-event associations

• Signal detection combines numerical data screening, statistical interpretation, and clinical judgement

• Most apparent associations represent known problems

• ~ 25% may represent signals about previously unknown associations

• The actual false positive rate is unknown

September 29, 2006

29

What Next?

• PhRMA/FDA working group has published a white paper addressing many of these issues

Drug Safety (2005) 28: 981-1007• Further refine methods, look for associations among

combinations of drugs and events, timing of reports • Data mining is like screening, need to evaluate

diagnostic properties of various approaches

• Need good dictionaries: many synonyms difficult signal detection

º Event names: MedDRA may help

º Drug names: Need a common dictionary of drug names to minimize dilution effect of synonyms

September 29, 2006

30

Data Used to Construct Plot

Intussception + Intussception -

Observed

Expected

Observed

Expected

RV +

DTAP +

85 74 1111 1122

DTAP - 29 40 608 597

RV - DTAP +

4 15 33520 33509

DTAP - 293 282 610714 610725