Doug Altman - MedicReS World Congress 2012

Tools for Assessing Risk of Bias

Doug AltmanCentre for Statistics in Medicine

University of Oxford

Acknowledgement

Cochrane Collaboration Bias Methods Group

Systematic review

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria to answer a specific research question

It uses explicit, systematic methods that are selected with a view to minimising bias, thus providing reliable findings from which conclusions can be drawn and decisions made

3

Key characteristics of a systematic review

A clearly stated set of objectives with an explicit, reproducible methodology

A systematic search that attempts to identify all studies that would meet the eligibility criteria

An assessment of the validity of the findings of the included studies, such as through the assessment of risk of bias

Systematic presentation and synthesis of the characteristics and findings of the included studies

[Liberati et al, BMJ 2007]4

Taking account of study quality

Methodological quality of the primary studies may affect their results

The influence of study quality should routinely be examined– How?

Quality is difficult to define– design, conduct and analysis of a trial? – clinical relevance?– quality of reporting?– all of these?

Focus on Randomised Controlled Trials (RCTs) 5

Quality scales and checklists

Many exist (>40)– vary hugely in complexity and scope

Many combine trial quality with reporting quality and other aspects (e.g. sample size)

Very popular: Jadad et al 1996– Simple score from 0 to 5– Serious flaws: combines methodology and reporting

Comparison of high vs low quality trials depends – Results depend strongly on which scale is used

[Jüni et al, JAMA 1999]

6

The Cochrane Collaboration aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions (and diagnostic test accuracy)

7

Cochrane library

8

Rethinking the approach to assessing study quality

2005: Cochrane Collaboration initiative

Rationale: quality scores are not helpful

Focus should be on aspects with potential for bias, such as – concealment of treatment allocation– blinding of outcome assessment– handling of patient attrition

9

Cochrane Risk of Bias (RoB) tool

The Risk of Bias (RoB) tool was developed over several years as a way to address concerns about the existing method of assessing study “quality” in Cochrane reviews

Based on empirical evidence where possible

It is intended to lead to uniform approach across reviews from all Cochrane review groups

10

Developing a Risk of bias tool

In May 2005, 16 statisticians, epidemiologists, and review authors attended a three day meeting to develop the new tool

Identified items that were truly potential biases rather than sources of heterogeneity or imprecision

Seven areas were discussed leading to a proposed set of criteria for assessing protection from bias – as adequate, inadequate, or unclear

11

The Risk of Bias Tool 

12

http://www.cochrane-handbook.org/

13

BMJ 2011;343:d5928

Risk of biasGeneral principle of the RoB tool Avoidance of bias

Assessment of risk of specific types of bias  

Selection bias

Performance bias

Detection bias

Attrition bias

Risk of bias summary14

15

Participants

Treatment A Comparator B

OutcomeAssessment

OutcomeAssessment

Detection bias

Selection bias

Attrition bias

Bias

Performance bias

16

Population

Treatment A Comparator B

OutcomeAssessment

OutcomeAssessment

Detection bias

Selection bias

Attrition bias

Bias

Performance bias

Randomisation

Blinding of participants and 

personnel

Blinding of outcome assessors

Domains of the Cochrane “Risk of Bias” tool

Review authors’ judgement: Was the allocation sequence adequately

generated? Was allocation adequately concealed? Was knowledge of the allocated intervention

adequately prevented during the study?– Participants– Trial personnel

Were incomplete outcome data adequately addressed?

Are reports of the study free of suggestion of selective outcome reporting?

Was the study apparently free of other problems that could put it at a high risk of bias?

17

Domains of the Cochrane “Risk of Bias” tool

Review authors’ judgement: Was the allocation sequence adequately

generated? Was allocation adequately concealed? Was knowledge of the allocated intervention

adequately prevented during the study?– Participants– Trial personnel

Were incomplete outcome data adequately addressed?

Are reports of the study free of suggestion of selective outcome reporting?

Was the study apparently free of other problems that could put it at a high risk of bias?

18

Separate assessment for each outcome

The “Risk of Bias tool” (RoB)General principles

19

2 steps What was reported

Extracted text: what was reported in the published report / protocol/ contact with authors

Commentary

Judgment relating to the risk of bias  Low risk of bias High risk of bias Unclear (judgment is impossible)

The “Risk of Bias tool” (RoB)General principles. What was reported?

Sequence generation. Low risk

Quote: “patients were randomly allocated”.Comment: Probably done, since earlier reports from the same investigators clearly describe use of random sequences (Cartwright 1980).

Blinding of participants and personnel (performance bias)

Low risk

Quote: “double blind, double dummy”; “High and low dose tablets or capsules were indistinguishable in all aspects of their outward appearance. For each drug an identically matched placebo was available”.Comment: Probably done

Blinding of outcome assessment (detection bias) (Mortality) 

Low risk

Quote: “Obtained from medical records”Comment: review authors do not believe this will introduce bias.

etc

20

The “Risk of Bias tool” (RoB)General principles. Judgment

High risk of bias Bias of sufficient magnitude to have a notable impact on the results

Unclear risk of bias Insufficient details reported Appropriate reporting, but the risk of bias is unknown

21

Attrition bias: Considerations

How much data is missing from each group?

Why is data missing in each group?

How were data analysed? Handling of incomplete outcome data

22

Attrition biasLow risk of bias• No missing outcome data

• Reasons for missing data not related to outcome

• Missing data balanced across groups, with similar reasons

• Missing data not enough to have a clinically relevant impact on the intervention effect estimate

• Missing data have been imputed using appropriate methods

23

Attrition biasHigh risk of bias• Reason for missing data related to outcome, with either imbalance in numbers or reasons

• Missing data enough to induce clinically relevant bias in intervention effect estimate

•  ‘As-treated’ analysis with substantial departure of the intervention received from that assigned at randomization

• Inappropriate use of imputation24

The “Risk of Bias tool” (RoB)General principles

Reviewers specifically trained

Independent duplicate assessment with consensus

Decisions need to be pre-specified in the protocol Classification of outcomes (subjective / objective) Blinding: successful blinding procedure Missing data Other risk of bias

Contact authors for missing information25

Selective outcome reporting

Concern is that there is a tendency to selectively report outcomes that are statistically significant– Empirical evidence [Kirkham et al, BMJ 2010]

Challenges in detecting such behaviour

Ideally need study protocol May be internal evidence from publications or

trials register Usually have to assess this item as “Unclear”

26

Risk of bias summary – each study

27

Risk of bias summary – all studies

28

Summary assessment of the risk of bias across trials

ClassificationLow risk of bias Most information is from trials at low risk of biasUnclear risk of bias Most information is from trials at low or unclear risk

of biasHigh risk of bias The proportion of information from trials at high

risk of bias is sufficient to affect the interpretation of results

29

Risk of bias for non-randomised studiesNonrandomised studies of interventions: Many issues are the same Main concern is selection bias – how were treatment

chosen? High risk of non-comparable groups

Some tools exist – e.g. “Newcastle-Ottawa” Cochrane Collaboration is planning to develop a RoB

tool

Other observational studies – e.g. prognosis, aetiology: Parallel developments are needed

30

Conclusions Assessing the risk of bias is an essential step for an appropriate interpretation of systematic reviews and meta-analysis

Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs Need for transparency Training and use of the handbook recommendations

Similar approach should be adopted for other study types

31

Closing comments

Good systematic reviews are a major source of reliable information on the benefits and harms of health care interventions

A key aspect is assessment of risk of bias of the primary studies

Cochrane Risk of Bias tool includes 7 items to be evaluated for RCTs– A similar approach is needed for other study types

Should take account of risk of bias in the analysis and interpretation

32

33