DSRU June 2011 1

An agency of the European Union

Defining objective measures of signal

prioritisation - issues to consider in the

masking effect of measures of

disproportionality François MAIGNEN (nothing to get hung about …)

Presented by: Name SurnamePosition or Unit/Sector/Section/Team

2

Plan of the presentation

Conflicts of interest

Background and rationale for the projects

Definition of objective measures of signal prioritisation

Assessment of the extent and magnitude of the masking effect of the PRR in EudraVigilance

Concluding remarks and take home messages

3

Conflict of interests

M. Hauben has contributed to this study

No external funding was received for this study

I do not have any financial interests with the Pharmaceutical industry or any IT software provider (declaration available from the Agency)

Disclaimer on the views expressed in this presentation wrt European Medicines Agency

No claim on a “better” safety profile on any medicinal product mentioned in this work should be made.

My declaration of interests can be obtained from the Agency

4

Acknowledgements

No external source of funding was used to perform this study. The following authors: FM has no conflicts of interest with the pharmaceutical industry (declaration of interest available from EMEA). M. Hauben is also working in Department of Medicine, Risk Management Strategy, Pfizer Inc., New York, New York University School of Medicine, Departments of Community and Preventive Medicine and Pharmacology, New York Medical College, Valhalla, New York, USA and for the School of Information Systems, Computing and Mathematics, Brunel University, London, England . None of the authors have any conflict of interests with any statistical software provider. M. Hauben did not (could not) access the raw data for confidentiality reasons.

5

Disclaimer

The views expressed in this presentation are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.

6

Signal prioritisation and serious medical events: reported rate of fatality as a prioritisation variable

About the EV-EWG IME list and lists of IMEs in general (e.g. CIOMS V)

Useful but purpose not always clear (early signal detection? Focus the detection? Signal prioritisation?)

Based on expert’s judgment

Has not been formally “validated” / tested (no standards)

Probably situation dependant

7

Concept of seriousness # linked to the outcome # surrogate for grading the severity of the reactions hence prioritisation

Grading in seriousness: death >> disability (permanent) >>> life-threatening >>> disability (temp.) >> prolongation hosp.

Variable linked to fatal outcome = reported rate of fatality

For each drug-event pair = No of reported fatal cases / total number of reported cases

Computed for the intensively monitored products

Reaction 1 Reaction 2 Reaction 3 Outcome (incl. fatal)

Surrogate to predict the outcome

8

Hazardous identification of serious events a priori

Some examples of reactions not usually considered to be serious per se which can be linked to most dramatic outcomes (e.g. dramatic increases of liver aminotransferases e.g. >100 ULN leading to liver failure, liver transplant and death)

Exhaustion/tiredness

Jaundiceincr. aminotransferases 500ULN

hyperbilirubinemia

Liver transplant Death

Prioritise these events on the associated reported outcome (here death)

9

Reported rate of fatality

Some reactions may be consistently linked to a high reported mortality rate

Some reactions are serious but do not lead to a fatal outcome

Some reactions are situation dependent (the reported rate of fatality may be highly variable)

For each of the MedDRA PT involved in a DEC in EudraVigilance, the following variables were computed across all the products involved in the reported combinations:

• Mean, min., max., range: max. – min., SD

10

Methods

DECs from last year in EudraVigilance concerning the “intensively” monitored products

Imported in ACCESS, computation of reported rate for each drug-event combination

Computation of average, min, max, SD and range for each MedDRA PT

Analysis in R using the ODBC connectivity

11

Average reported rate of fatality

Significant No of terms associated with a very low rate or no fatality (0%)

12

How does it relate to IME status?

Important No of IME terms not associated with fatal outcome (or associated with a very low reported rate)

RHS: IMEs

LHS: non-IMEs

13

How does it relate to IME status?

Reported rate of fatality for IMEs > non-IMEs

Number of events for which the reported rate is high which are non-IMEs

Very high number of IMEs for which the reported rate of fatality is zero.

IMEs useful for prioritisation?

The figure displays the boxplot of the average reported rate of fatality for non-IMEs (left)compared to IMEs (right) (red and blue line = mean rate for non-IMEs (red) vs IMEs (blue))

14

Mean / median

Difference of the mean rate / median between non-IMEs and IMEs but not very important (approx. 10%)

Confirmation that some non-IMEs are linked to a fairly high reported rate of fatality

And that some IMEs are associated to a very low rate

The analysis shows that some non-IMEs have a high minimum rate (graph not shown)

The median of the max. reported rate of fatality is quite low.

Can IMEs discriminate between serious / non-serious signals?

15

Relation between the different variables?

Lattice of the descriptive variables for the reported rate of fatality

16

17

18

19

Liver injuries

Clear relation between Reported rate # seriousness of injury and the severity of the outcome

Highest mean rate around 30% (1/3 fatal reports) with a max. at 75% (3/4)

Some inconsistencies (bilirubin disorders: hyperbilirubinaemia 18.7%, blood bilirubin increased 16.2%, blood bilirubin unconjugated increased 6.7% and bilirubin conjugated increased 6.3%)

Unclear or undefined concepts (liver disorders [?]) linked to a fairly high mean reported rate 18.9%, hepatic function abnormal 8.5% and liver function test (singular) abnormal 9.1%.

20

Data reduction (PCA)

21

Data reduction (PCA)

Principal components analysis (sorry I can’t remember whether I did it on the variance-covariance or on the correlation matrix)

Performed on Average, Min, Max, Range

99% of variability explained by first two PCs +++

First two PC closely associated to Max and Average

22

DiscussionThree set of events used for signal detection: mild reported rate of mortality, moderate and high

Reported rate of fatality can be useful (and should be used) for signal prioritisation

Needs to be considered with caution (events with rate of zero include e.g. Torsade de pointes, autism, Breast cancer in situ, Breast cancer stage I, Dermatitis exfoliative)

Does not replace DMEs

Death is not the only criterion which could be used

EudraVigilance = only serious reactions(!)

Some events are consistently associated either with low rate or conversely with very high rate

An agency of the European Union

Assessing the effect and extent of the masking effect of the PRR on SRS databasesF. Maignen / M. Hauben

24

Credits and acknowledgements

M. Hauben dragged me into this project against my will (he should be considered to be the promoter). FM devised the mathematical background. MH came up with the idea to use the EU-ADRs IMEs. FM added the additional MedDRA terms relevant to Public Health.

None of the authors have any conflict of interests with any statistical software provider. M. Hauben did not (could not and still cannot) access the raw data for confidentiality reasons.

25

Masking effect of measures of disproportionality (here = PRR)

The masking effect has first been described and identified by Gould in spontaneous reporting system databases (pharmacoepidemiology and drug safety in 2003 – 8 years ago).

The masking is a statistical artefact by which true signals are hidden by the presence of information reported with other medicines in the database. Therefore, the masking involves one given reaction and two products (the product for which the DA is conducted) and a possible masking drug.

The masking effect is a potentially important issue for Public Health which is not perfectly understood or perfectly quantified: some signals might be missed or identified with delay because of the presence (or a suspicion on the presence) of masking effect.

26

Masking effect of measures of disproportionality

In particular, there is no algorithm to identify the potential masking drugs to remove them from subsequent analyses aimed at identifying new signals using the statistical methods of signal detection based on disproportionality analysis.

We have developed an algorithm based on the computation of a simple The masking ratio has been developed to be intuitive. The highest masking drugs have the highest masking ratio.

From an underlying mathematical framework, we have developed a simple expression of the masking ratio (which can be easily computed on a database incl. No of computations and IT resources) which allow a fairly rapid identification of the main culprits.

27

28

Masking effect of measures of disproportionality

Recent studies have shown effects which were suspected from the article by Gould, that masking products are usually products for which the given reaction is known (i.e. listed in the SPC), therefore likely to have a high PRR (in the database in which the analysis is conducted) for the adverse drug event / reaction which is included in the disproportionality analysis.

Unfortunately, the authors could not conclude on any algorithm considering that this association is not systematically present (not all products with high PRR will induce a significant masking even if he masking generally involves products with a high PRR).

29

Masking effect of measures of disproportionality

The removal of the masking effect has the advantage to put the focus on a defined threshold (the signal is below or above the chosen threshold) rather than on the ranking.

Both approaches should provide a clear understanding how the removal of the masking product affects the ranking provided by the different methods of disproportionality.

30

Masking effect of measures of disproportionality (RRR)

Respective proportion of reports in the

database influences the extent of the

masking

The higher the proportion of reports

involving a product for a given

reaction the higher the masking

The lower the proportion involving a

given product over the total number of

reports in the database, the higher the

masking

31

Masking effect of measures of disproportionality

A reassuring result shows that if the masking is fairly prevalent in a database (approx. 60% of DECs have a MR > 1), however, the prevalence of a significant masking (MR > 1.1) is quite rare (0.006% of the DECs) and mainly affects events rarely reported in the database.

Some assumptions of size of the database must be checked when the model is implemented (our model is likely to be valid without any further assumptions of size) in EudraVigilance or SRS databases of a similar size.

32

33

Relation between the masking effect and the PRR (of the masking medicinal product for the given event)

MR > 1PRR > 2

34

Relation between the masking effect and the PRR (of the masking medicinal product for the given event)

MR > 1PRR > 2

35

The highest masking is induced by products known to induce the given reaction (and for which the PRR is likely to be increased)

Products of the same class induce the highest masking for similar reactions (gambling – ropinirole and pathological gambling – cabergoline, Fanconi syndromes, role of drug-drug interactions – rifampicin)

36

Exact MR

Requires an important number of cumbersome computations (easy to make a mistake in the contingency tables) since the exact MR:

-Is specific of both the DEC being masked and of the masking product, therefore the exact MR requires n*(n – 1) computations (where n is the No of DECs)

The simplified MR relies on approximations related to the size of the database (verified but it needs to be checked).

37

Simplified expression of the MRSimplified masking ratio is perfectly correlated to the exact MR

The simplified MR slightly overestimates the exact MR (not more than 10%)

Provides an excellent variable for an algorithm

But more importantly, the removal of the masking effect of the highest masking product marginally affects the ranking given by the measure of disproportionality (based on EXACT masking ratio)

The algorithm can be extended to more than one product since, by construction, the MR is multiplicative (caveat: MRC/A must be computed on new contingency table without B)

PRRA(withoutB)(withoutC)=MRB/A*MRC/A*PRRA(incl. B and C)

38

Simplified expression of the MR

39

The ranking is marginally affected by the removal of the masking effect induced by the highest masking drug (exact MR) (Ab anti Epo)

More signals

Ranking MARGINALLYaffected

40

The ranking is marginally affected by the removal of the masking effect induced by the highest masking drug (exact MR) (liver failure)

Ranking NOTaffected

41

Pitfalls•Overlapping datasets: the removal of the masking must be performed adequately (only remove the records which involve the masking product)

•Masking is different from SUBGROUPING (different algorithm which also relies on assumptions on the size of the db)

•The method of computation of the DA (report level vs event level) is likely to influence the masking (magnitude, removal of the records)

•Masking possibly more prevalent in Companies databases rather than Regulatory Authorities dbs (prevalence and highest masking effect)

•Assumptions on the size of the database must be checked in particular in small SRS databases

42

Proposed algorithm

43

Conclusions

Rationale approach to signal prioritisation and identification of the masking effect of measures of disproportionality

Simplified variable from a rather cumbersome mathematical framework

Pitfalls and caveats must be kept in mind, in particular the assumptions on the size of the database must be checked, Masking is different from subgrouping, overlapping datasets must be handled appropriately

The prevalence of significant masking (>10%) is rather low. The masking is induced by products which themselves are known to induce the reaction for which they are responsible of a masking effect

44

Conclusions

The masking marginally affects the ranking given by the measure of DA

The proportion of true signals / false positive revealed by the masking is currently unknown

The median time gained by the removal of the masking effect is also unknown

The analyses are exploratory in nature and one should not deviate from the general consensus concerning the systematic evaluation of SDRs

Masking possibly more prevalent in Companies SRS dbs

45

Thank you!!

Comments to:

[email protected]