Focus – Benefit–risk assessment and frameworks … · Political Science (LSE); Francesco Pignatti, EMA; Xavier Luria, EMA. Keywords ... intended for training and initial implementation.

5

AuthorsNikolaos Zafiropoulos, European Medicines Agency (EMA); Lawrence Phillips, EMA and London School of Economics and Political Science (LSE); Francesco Pignatti, EMA; Xavier Luria, EMA.

KeywordsEuropean Medicines Agency (EMA); Committee for Medicinal Products for Human Use (CHMP); Marketing authorisation; Regulatory decision-making; benefit–risk assessment (BRA); Multi-Criteria Decision Analysis (MCDA); Work packages.

AbstractIn order for a medicinal product to get marketing authorisation in Europe, it has to demonstrate a positive benefit–risk balance. Although this has been the cornerstone of the evaluation process, there is no standardised methodology that is used in this context. Recognising the need for a structured approach that can enhance the transparency and consistency in assessing the benefit–risk balance, the European Medicines Agency (EMA) began a three-year project in early 2009. This project consists of five consecutive work packages. The first four work packages form a research phase that aims to develop and test tools and methods for balancing benefits and risks of medicinal products. The fifth work package is intended for training and initial implementation. Currently, the project has completed its research phase, which recognised the usefulness of two levels of gradation. The first level is a qualitative approach, mainly consisting of a table listing the key effects of the benefit–risk balance and their uncertainty in a common and concise format. The second level, recommended for more complex situations, is a quantitative method that utilises Multi-Criteria Decision Analysis (MCDA) to derive a numerical value for the benefit–risk balance. However, the implementation of MCDA in the assessment of a medicine poses some practical challenges that remain to be addressed in the last work package of the project.

IntroductionThe three pillars of regulatory approval for new pharmaceutical products are quality, safety and efficacy. In order for a new drug to access the European market through the centralised procedure, it has to demonstrate compliance to manufacturing standards, an acceptable safety profile for its indication and a proven effect on specific efficacy parameters. On top of this structure is the benefit–risk balance, which has to be positive in order for the product to get marketing authorisation. European legislation defines the benefit–risk balance as an evaluation of the positive therapeutic effects of the medicinal product in relation to its risks as regards patients’ health or public health.1

Assessing the benefit–risk balance can be a challenging process involving the evaluation of complex, or even controversial, sets of data.

Evaluating benefit–risk: An Agency perspective

In addition, there is always a degree of uncertainty around the actual benefits and risks of a medicine since they can only be determined by assessing the information provided. This process mainly involves extensive discussions at national level, at the agencies that have been appointed as rapporteurs of a given product, with input from advisory committees and field experts. It is also complemented by peer reviewing among European agencies, before there is a discussion and agreement at the level of the EMA’s Committee for Medicinal Products for Human Use (CHMP). Although the validity of this process was never questioned, the lack of a certain degree of consistency and transparency has been recognised. Noticeably, there has been no detailed guidance on the principles and methodology for benefit–risk assessment, and none of the main regulatory authorities (EU, US, Japan) has issued a list of benefit and risk criteria. Contrary to the paramount need for a systematic and structured framework for assessing the benefit–risk balance, there has not been any substantial change to the currently established paradigm. In an effort to address this gap, the EMA’s CHMP agreed to start research in the methodology of the benefit–risk assessment.2

As a result, the EMA began the Benefit–Risk Methodology Project in 2009 with the aim to explore methodologies that can increase the consistency and transparency of the benefit–risk assessment for medicinal products. The national competent authorities of France, Germany (Paul-Ehrlich-Institut), the Netherlands, Spain, Sweden and the UK volunteered to participate. Experts in the field of Decision Theory were engaged to provide scientific leadership and support. The project consists of five consecutive work packages. The first four work packages form the research phase of the project and the fifth is intended for training and initial implementation. The project foresees also a public consultation phase and a workshop to engage relevant stakeholders.3 Currently, the project has completed the four work packages that form the research phase.

The aim of the first work package was to explore the current practice of benefit–risk assessment. In this context, the project team was invited to visit the participating agencies and interview a number of key staff members. The team also attended a number of internal meetings and discussions. The purpose of the visits was observational, with an aim to extract an overall view of how benefit–risk assessment is done Europe-wide, without interfering with the agencies’ procedures. The main finding was that the benefit–risk balance is done mainly intuitively based on expert judgment, without the use of a systematic approach.4

Work package 2 was a review of methods that have appeared in the scientific literature and could potentially support the benefit–risk assessment of medicines. The project team reviewed in total four qualitative frameworks and 18 quantitative approaches, and assessed their applicability in the context of the centralised procedure. Decision Theory was found to be the most appropriate basis for quantifying favourable and unfavourable effects on a common scale, including their clinical relevance and associated uncertainties. MCDA was recognised as the most relevant methodology in this context. It was also suggested that combinations of methods could be useful in more complex situations.5

Focus – Benefit–risk assessment and frameworks

www.topra.org Regulatory Rapporteur – Vol 9, No 6, June 2012

6

In the next phase, work package 3, the project team visited five of the participating agencies in order to field test the most appropriate methods. The main finding was that quantitative modelling of the benefit–risk assessment is feasible and could improve the whole process, but might not always be necessary. For this reason, two levels of gradation were found useful, a qualitative and a quantitative, depending on the complexity of the benefit–risk data to be assessed.6,7

The objective of work package 4, the final part of the research phase of the project, was to recommend a methodology or a set of methods for supporting the benefit–risk assessment, based on the findings of the previous work package. A basic recommendation by the project team was the use of a general framework, the PrOACT-URL,8 as a guide for structuring the thinking process. From a methodological perspective, two levels were suggested by the team depending on the complexity of the benefit–risk data to be assessed. The first level is a qualitative approach, consisting of a table of effects and their uncertainties. This format allows visualisation of the key effects in a simple and concise manner. For more complex situations, eg, multiple conflicting effects, a second level is proposed, based on MCDA. The final recommendation is the use of specialised software for generating graphical displays that can support the representation and discussion of the final results.9

Work package 5 is currently ongoing, and is intended for training of the rapporteurs and assessors, and for initial implementation of the new methodology.

Current practice of benefit–risk assessment4

During the initial phase of the project, the project team visited the six participating agencies (France, Germany (Paul-Ehrlich-Institut), the Netherlands, Spain, Sweden and the UK) in order to explore the current practice of evaluating and balancing benefits and risks, and to obtain a better understanding of each agency’s decision-making process. The approach of the team was to interview staff members with a key role in the benefit–risk assessment. In total, 55  staff members were interviewed including assessors, statisticians and the EMA’s CHMP members. All interviews followed a protocol, developed by the team

Table 1: The eight summary headings of the interview protocol.

Interview protocol

1 Agency’s history and purpose2 Agency’s relationships with governmental and non-

governmental organisations3 Agency’s organisational structure4 Information flow5 Meaning of “benefits” and “risks”6 Benefit–risk assessment process7 Consistency8 Existence of models.

Table 2: The four-fold frame for separating the elements of the benefit–risk assessment.

Four-fold benefit–risk frame

Favourable effects Uncertainty of favourable effects

Unfavourable effects Uncertainty of unfavourable effects

in advance, which consisted of a list of questions under eight summary headings (see Table 1). The team was also invited to attend a number of internal meetings and discussions. The visits were observational without interfering with the agencies’ procedures.

The team noticed that interviewees, both between agencies and within the same agency, expressed divergent views on the meaning of benefits and risks, and on their weighing. Especially with regard to risks, there was a great variance of responses. Most notably, the uncertainty of realising a benefit from the use of a pharmaceutical product was frequently mentioned as a risk. Additionally, in all the six agencies there was no mention of the use of a validated and structured process for assessing the benefit–risk balance. In contrast, most of the interviewees acknowledged the need for a more systematic approach and the value it can add.

The main recommendation from this work package was the introduction of a clear separation between the type of effects that derive from the use of a pharmaceutical product, and the uncertainty surrounding these effects. In this context, the project team suggested the use of a four-fold frame, presented as a table (see Table 2). Each element of the benefit–risk analysis should have a unique and universal classification based on this frame.

Prior to forming a benefit–risk balance, the elements pertaining to the frame should be accordingly separated to each cell. This could help structure the assessment and facilitate communication and transparency of the regulatory decision. The benefit–risk section of the EMA’s CHMP assessment report template-guidance was revised in September 2009 to incorporate the four-fold frame.

Applicability of current tools and processes5

After acquiring an understanding of the current practice, the project team reviewed several methods that have appeared in the scientific literature. The aim was to assess their applicability and usefulness for benefit–risk assessment based on the experience gained from the previous phase. The project team identified four qualitative frameworks and 18 quantitative approaches that appeared relevant. (A qualitative approach provides structure to the benefit–risk assessment without requiring substantial processing of the numerical values of the benefits and risks. In contrast, a quantitative approach has a mathematical component that involves a numerical transformation of the benefits and risks values.)

The first consideration was that the basis for any structured approach to benefit–risk assessment is a decision framework. A generic framework, the PrOACT-URL,8 which provides a generic decision problem structure, was adapted for regulatory use as presented in Table 3.

Only three quantitative approaches were considered sufficiently comprehensive to enable the benefit–risk assessment to be represented numerically (as a difference or a ratio) by incorporating the value or utilities of favourable and unfavourable effects, along with probabilities representing the uncertainties of those effects. These were Bayesian statistics, Decision Trees and MCDA. More specifically, Decision Trees are useful in situations where uncertainty is the main issue and MCDA when there is a need to compare conflicting criteria.

Five other approaches, while more restricted in scope, could be useful for particular cases: probabilistic simulation when the focus is on uncertainty of effects; Markov processes and Kaplan-Meier estimators for changes in health states over time; QALYs for modelling multiple health outcomes; and conjoint analysis for eliciting patient preferences and trade-offs.

It was also recognised that a combination of approaches could be needed in more complicated situations.

Field tests6,7

After recognising a set of methods that could support the benefit–risk assessment, the team aimed at field testing those that would prove more relevant. For each field test, the PrOACT-URL framework (Table 3) was applied as a general structure. An Effects Table of the main criteria was

Focus – Benefit–risk assessment and frameworks

Regulatory Rapporteur – Vol 9, No 6, June 2012 www.topra.org

7

created that provided definitions, the data, the units as well as upper and lower limits of the scoring scales. Table 4 shows a hypothetical example of an Effects Table for Caprelsa (Zictifa), one of the drugs modelled.

During the preparation for the five field tests, it became apparent for which specific cases MCDA would be the most relevant approach in order to develop a comprehensive model. The main reason was that all the five cases were characterised by the need to compare dissimilar favourable and unfavourable effects (eg, progression-free survival vs QTc prolongation). The two key points of an MCDA model are:1 Each favourable and unfavourable effect to be compared is

converted to a preference value on a 0–100 scale. This conversion is accomplished by a linear or non-linear translation, called a ‘value function’, which is an assessment of the clinical relevance of the effect

2 The units for the preference value scales are equated through a process known as ‘swing weighting’, which requires judgments of the clinical relevance of scale differences. This enables weighted effects to be summed to give an overall benefit–risk balance that is based on both the evidence and its clinical relevance.Following these steps, relevant graphical displays were generated

representing the benefit–risk balance. Figure 1 shows a hypothetical example of the added-value graph in the case of Caprelsa. This graph presents the overall score in terms of preference value for the drug and the placebo. Longer green bars represent more benefit while longer red bars indicate less risk.

In order to check the robustness of the results, the team would perform sensitivity analysis on the various weights of several criteria. This type of analysis is useful for exploring uncertainties and divergent views on the importance (assigned weight) of a criterion. Another way to check the robustness of the results is with scenario analysis, in which the team would explore various assumptions on the performance of a criterion. For example, the team would rerun the analysis using the values of the upper or lower limit of the confidence intervals. This is relevant for assessing the impact of the uncertainty of the data.

It was generally acknowledged that this type of methodology can easily test different perspectives for their impact on the benefit–risk balance and that it helps explore the effect of uncertainty.

Proposed benefit–risk tools and methods9

The aim of the fourth work package was to recommend methodologies that can support the benefit–risk assessment. Based on the experience from the field tests, and recognising the need to accommodate the perspectives of the various national competent authorities in a resource-efficient manner, the project team recommended a gradated methodology. It was recognised from the previous step that quantitative modelling can be superfluous in more simple cases. In this context, the project team made four recommendations, assuming that the assessment is not so obvious that no help at all is needed.

As a basic principle the project team recommended PrOACT-URL (Table  3), a general decision framework, which can support the thinking process of the benefit–risk assessment. It is not necessary for this to be prepared in writing; the intention is to provide a general guide of the main points that should be considered. This can be used irrespective of any other tool or method.

The next recommendation was the incorporation of the Effects Table (Table  4) to the EMA’s CHMP assessment reports. The table displays all the favourable and unfavourable effects that were considered as influencing the benefit–risk balance, along with definitions of the effects, the unit of measurement, the plausible range of data, the measured data (pooled or separately for each clinical trial) associated with the medicinal product and any comparators, including confidence intervals where appropriate. An additional column could be added in order to include comments about remaining uncertainties and concerns. This representation of the main contributors to the benefit–risk assessment in a common and concise format would facilitate the readability and understanding of the assessment.

The third recommendation was the use of MCDA for more demanding assessments. If a linear ‘value function’ can be assumed for the conversion of each effect to a preference scale and extensive

Table 3: A summary of the eight-step PrOACT-URL framework as adapted for regulatory use.

1 Problem Determine the nature of the problem and its context

2 Objectives Establish objectives and identify criteria of favourable and unfavourable effects

3 Alternatives Identify the options to be evaluated against the criteria

4 Consequences Describe how the alternatives perform for each of the criteria

5 Trade-offs Assess the balance among favourable and unfavourable effects

6 Uncertainty Assess the uncertainty associated with the effects

7 Risk tolerance Judge the relative importance of the decision-maker’s risk attitude

8 Linked decisions

Consider the consistency of this decision with past/future decisions.

Focus – Benefit–risk assessment and frameworks

www.topra.org Regulatory Rapporteur – Vol 9, No 6, June 2012

Note: Overall benefits are shown as the upper green bars, while the lower red bars indicate less risk.

Figure 1: The overall weighted score for Caprelsa (Zictifa) and placebo, 77 and 74 respectively, showing the first as the preferred option.

Favourable Effects/Unfavourable Effects (FE/UFE) Balance

FE

UFE

TOTAL 359 77 74 100.0

40.9147

212 59.1

WeightWeight

Zictifa Placebo Cumulative

8

Table 4: The Effects Table for Caprelsa (vandetanib, treatment of inoperable thyroid cancer).

Effect Description Best Worst Units Placebo 300 mg

Favo

urab

le e

ffec

ts Prim

ary

endp

oint Progression-

free survival Hazard Ratio

Date of randomisation to the date of objective progression or death (blinded independent review)

0 1 unitless 1 0.46

Seco

ndar

y en

dpoi

nts

Progression-free survival (median)

Date of randomisation to the date of objective progression or death (Weibull model)

60 0 months 19.3 30.5

Objective Response (RECIST)

Proportion of complete or partial responders (at least a 30% decrease in the sum of the longest diameter of target lesions compared to baseline)

100 0 % 13 45

Unf

avou

rabl

e eff

ects

Diarrhoea, Grade 3-4

Increase of ≥7 stools per day over baseline; incontinence; IV fluids ≥24 hrs; hospitalisation; severe increase in ostomy output compared to baseline; interfering with activities of daily living; Life-threatening consequences

0 100 % 2.0 10.8

QTc related events, Grade 3-4

QTc >0.50 second; life-threatening signs or symptoms (eg, arrhythmia, CHF, hypotension, shock syncope); Torsade de pointes

0 100 % 1.0 13.4

Infections, Grade 3-4

IV antibiotic, antifungal, or antiviral intervention indicated; interventional radiology or operative intervention indicated; life-threatening consequences.

0 100 % 36.4 49.8

sensitivity analysis is not required, MCDA can be performed in a simple manner without the use of specialised software. It will still provide a numerical value for the benefit–risk assessment and can serve as a platform for basic scenario analysis.

The final recommendation was the utilisation of effective graphical displays and extensive sensitivity analysis through the support of specialised MCDA software. In the case of very complicated assessments, where, for example, there are non-linear value functions and there is a need for extensive sensitivity analysis, graphical displays similar to Figure 1 can be presented in order to support the outcome of the benefit–risk assessment.

Training and initial implementationThe final part of the research phase of the project (ie, work package 4), was finalised in February 2012 and the project has effectively entered work package 5, which aims at training and initial implementation. The implementation of MCDA in the assessment of a medicine throughout its lifecycle poses some practical challenges in the networked environment of the EU. The last work package of the project is expected to address some of these challenges.

Depending on the outcome of this current phase and with the agreement of the EMA’s CHMP, a public consultation will be launched on the first four work packages that will also include a workshop to engage the project’s stakeholders. Following that, EMA will adopt a final position.

DisclaimerThe views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties. References to products are for illustration purposes only, and not intended to be an accurate reflection of the data submitted. The information in this article was not taken into account by the EMA’s CHMP during the scientific assessment of marketing authorisation applications.

References1 Directive 2001/82/EC of 6 November 2001. http://ec.europa.eu/health/files/

eudralex/vol-1/dir_2001_83_cons/dir2001_83_cons_20081230_en.pdf

2 ‘Reflection paper on benefit-risk assessment methods in the context

of the evaluation of marketing authorisation applications of medicinal

products for human use’ (EMEA/CHMP/15404/2007), 19 March 2008.

http://www.ema.europa.eu/docs/en_GB/document_library/Committee_

meeting_report/2009/10/WC500006118.pdf

3 ‘Benefit-Risk Methodology Project’ (EMEA/108979/2009), 12 March

2009. http://www.ema.europa.eu/docs/en_GB/document_library/

Report/2011/07/WC500109477.pdf

4 ‘Benefit-risk methodology project: Work package 1 report: Description of

the current practice of benefit-risk assessment for centralised procedure

products in the European Union regulatory network’ (EMA/227124/2011),

25 May 2011. http://www.ema.europa.eu/docs/en_GB/document_library/

Report/2011/07/WC500109478.pdf

5 ‘Benefit-risk methodology project: Work package 2 report: Applicability

of current tools and processes for regulatory benefit-risk assessment’

(EMA/549682/2010), 31 August 2010. http://www.ema.europa.eu/docs/

en_GB/document_library/Report/2010/10/WC500097750.pdf

6 ‘Benefit-risk methodology project: Work package 3 report: Field tests’

(EMA/718294/2011), 31 August 2011. http://www.ema.europa.eu/docs/

en_GB/document_library/Report/2011/09/WC500112088.pdf

7 L D Phillips et al. ‘Is quantitative benefit–risk modelling of drugs desirable

or possible?’, Drug Discovery Today: Technologies, 8(1),ppe3–e10, 2001.

8 J S Hammond, R L Keeney, H Raiffa. ‘Smart Choices: A Practical Guide to

Making Better Decisions’, Harvard University Press, Boston, 1999.

9 ‘Benefit-risk methodology project: Work package 4 report: Benefit-risk tools

and processes’ (To be published on the EMA website). http://www.ema.

europa.eu/docs/en_GB/document_library/Report/2012/03/WC500123819.pdf

Focus – Benefit–risk assessment and frameworks

Regulatory Rapporteur – Vol 9, No 6, June 2012 www.topra.org