ppt - Slide 1

RATING QUALITY OF EVIDENCE AND STRENGTH OF RECOMMENDATIONS IN

GI USING THE GRADE FRAMEWORKAGA Clinical Practice & Quality Management Committee

Teleconference 17 Oct 2008

Yngve Falck-Ytter, M.D.Assistant Professor of Medicine

Case Western Reserve University, ClevelandDivision of Gastroenterology, Case and VA Medical Center

Disclosure

In the past 4 years, Dr. Falck-Ytter received no personal payments for services from industry. His research group received research grants from Valeant and Roche that were deposited into non-profit research accounts. He is a member of the GRADE working group which has received funding from various governmental entities in the US and Europe. Some of the GRADE work he has done is supported in part by grant # 1 R13 HS016880-01 from the Agency for Healthcare Research and Quality.

Content

Background and rationale for revisiting guideline methodology

The GRADE approach

Grading the quality of evidence and strength of recommendations in GI

Hierarchy of evidence

STUDY DESIGN Randomized Controlled

Trials Cohort Studies and

Case Control Studies Case Reports and Case

Series, Non-systematic observations

BIAS

Expert Opinion

Exp

ert O

pin

ion

Expert Opinion

Reasons for grading evidence?

People draw conclusions about the quality of evidence and strength of

recommendations

Systematic and explicit approaches can help protect against errors, resolve disagreements communicate information and fulfill needs

Change practitioner behavior However, wide variation in

approachesGRADE working group. BMJ. 2004 & 2008

6

Grading used in GI CPGsAASLD AGA ACG ASGE

I RCTs I RCTs, well designed, n↑ for suff. stat. power

I Syst. review of RCTs

A. Prospect. controlled trials

II-1 Controlled trials(no randomization)

II 1 large well-designed clinical trial (+/- rand.), cohort or case-control studies or well designed meta-analysis

II-3 Multiple time series, dramatic uncontr. experiments

III Opinion of respected authorities, descrip. epidemiology

II-2 Cohort or case-control analytical studies

III Clinical experience, descr. studies, expert comm.

IV Not rated

II 1+ properly desig. RCT, n↑, clinical setting

III Publ., well-desig. trials, pre-post, cohort, time series, case-control studies

IV Non-exp. studies >1 center/group, opinion respected authorities, clinical evidence, descr. studies, expert consensus comm.

B. Obser-vational studies

C. Expert opinion

Limitations of existing systems

Confuse quality of evidence with strength of recommendations

Lack well-articulated conceptual framework

Criteria not comprehensive or transparent

GRADE unique breadth, intensity of development process wide endorsement and use conceptual framework comprehensive, transparent criteria

Focus on all important outcomes related to a specific question and overall quality

GRADE Working GroupDavid Atkins, chief medical officera Dana Best, assistant professorb Martin Eccles, professord Francoise Cluzeau, lecturerx

Yngve Falck-Ytter, associate directore Signe Flottorp, researcherf Gordon H Guyatt, professorg Robin T Harbour, quality and information director h Margaret C Haugh, methodologisti David Henry, professorj Suzanne Hill, senior lecturerj Roman Jaeschke, clinical professork Regina Kunx, Associate ProfessorGillian Leng, guidelines programme directorl Alessandro Liberati, professorm Nicola Magrini, directorn

James Mason, professord Philippa Middleton, honorary research fellowo Jacek Mrukowicz, executive directorp Dianne O’Connell, senior epidemiologistq Andrew D Oxman, directorf Bob Phillips, associate fellowr Holger J Schünemann, professorg,s Tessa Tan-Torres Edejer, medical officert David Tovey, Editory

Jane Thomas, Lecturer, UKHelena Varonen, associate editoru Gunn E Vist, researcherf John W Williams Jr, professorv Stephanie Zaza, project directorw

a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USAc) Centers for Disease Control and Prevention, USAd) University of Newcastle upon Tyne, UKe) German Cochrane Centre, Germanyf) Norwegian Centre for Health Services, Norwayg) McMaster University, Canadah) Scottish Intercollegiate Guidelines Network, UKi) Fédération Nationale des Centres de Lutte Contre le Cancer, Francej) University of Newcastle, Australiak) McMaster University, Canadal) National Institute for Clinical Excellence, UKm) Università di Modena e Reggio Emilia, Italyn) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italyo) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Polandq) The Cancer Council, Australiar) Centre for Evidence-based Medicine, UKs) National Cancer Institute, Italyt) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USAx) University of London, UKY) BMJ Clinical Evidence, UK

GRADE uptake

11

GRADE: Quality of evidence

The extent to which our confidence in an estimate of the treatment effect is adequate to support particular recommendation.

Although the degree of confidence is a continuum, we suggest using four categories:

High Moderate Low Very low

I B II V III

Quality of evidence across studies

Outcome #1Outcome #2Outcome #3

Quality: HighQuality: ModerateQuality: Low

Determinants of quality

RCTs start high

Observational studies start low

What lowers quality of evidence? 5 factors: Detailed design and execution Inconsistency of results Indirectness of evidence Imprecision Publication bias

14

What is the study design?

1. Design and execution

Study limitations (risk of bias) Lack of allocation concealment No true intention to treat principle Inadequate blinding Loss to follow-up Early stopping for benefit

2. Consistency of results

Look for explanation for inconsistency patients, intervention, comparator, outcome,

methods

Judgment variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2

Pagliaro L et al. Ann Intern Med 1992;117:59-70 17

Heterogeneity

3. Directness of Evidence

Indirect comparisons Interested in head-to-head comparison Drug A versus drug B Infliximab versus adalimumab in Crohn’s

disease

Differences in patients (early cirrhosis vs end-stage cirrhosis) interventions (CRC screening: flex. sig. vs

colonoscopy) outcomes (non-steroidal safety: ulcer on

endoscopy vs symptomatic ulcer complications)

4. Imprecision

Small sample size small number of events wide confidence intervals uncertainty about magnitude of effect

5. Reporting Bias (Publication Bias)

Reporting of studies publication bias

number of small studies Reporting of outcomes

21

Quality assessment criteria

Lower if…Quality of evidence

High (4)

Moderate (3)

Low (2)

Very low (1)

Study limitations(design and execution)

Inconsistency

Indirectness

Imprecision

Publication bias

Observational study

Study design

Randomized trial

Higher if…

What can raise the quality of evidence?

BMJ 2003;327:1459–61 22

23

Quality assessment criteria

Lower if… Higher if…Quality of evidence

High (4)

Moderate (3)

Low (2)

Very low (1)

Study design

Randomized trial

Observational study

Study limitations

Inconsistency

Indirectness

Imprecision

Publication bias

Large effect (e.g., RR 0.5)Very large effect (e.g., RR 0.2)

Evidence of dose-response gradient

All plausible confounding would reduce a demonstrated effect

24

Categories of quality

LowFurther research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

Further research is very unlikely to change our confidence in the estimate of effectHigh

ModerateFurther research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

Very low Any estimate of effect is very uncertain

25

Judgments about the overall quality of evidence Most systems not explicit

Options: Benefits Primary outcome Highest Lowest

Beyond the scope of a systematic review

GRADE: Based on lowest of all the critical outcomes

GRADE evidence profile

Strength of recommendation

“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”

Although the strength of recommendation is a continuum, we suggest using two categories :

“Strong” and “Weak”

Desirable and undesirable effects Desirable effects

Mortality reduction Improvement in quality of life, fewer

hospitalizations/infections Reduction in the burden of treatment Reduced resource expenditure

Undesirable effects Deleterious impact on morbidity, mortality or

quality of life, increased resource expenditure

Determinants of the strength of recommendation

Factors that can weaken the strength of a recommendation

Explanation

Lower quality evidence The higher the quality of evidence, the more likely is a strong recommendation.

Uncertainty about the balance of benefits versus harms and burdens

The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely is a weak recommendation warranted.

Uncertainty or differences in values

The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.

Uncertainty about whether the net benefits are worth the costs

The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted.

Developing recommendations

Implications of a strong recommendation

Patients: Most people in this situation would want the recommended course of action and only a small proportion would not

Clinicians: Most patients should receive the recommended course of action

Policy makers: The recommendation can be adapted as a policy in most situations

Implications of a weak recommendation Patients: The majority of people in this

situation would want the recommended course of action, but many would not

Clinicians: Be prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making

Policy makers: There is a need for substantial debate and involvement of stakeholders

Where GRADE fits inPrioritize problems, establish panel

Systematic review

Searches, selection of studies, data collection and analysis

Assess the relative importance of outcomes

Prepare evidence profile: Quality of evidence for each outcome and summary

of findingsAssess overall quality of evidence

Decide direction and strength of recommendation

Draft guideline

Consult with stakeholders and / or external peer reviewer

Disseminate guideline

Implement the guideline and evaluate

GR

AD

E

Systematic review

Guideline development

PICO

OutcomeOutcomeOutcomeOutcome

Formulate

question

Rate

importa

nce

Critical

Important

Critical

Not important

Create

evidence

profile with

GRADEpro

Summary of findings & estimate of effect for each outcome

Rate overall quality of

evidence across outcomes based

on lowest quality of critical outcomes

Panel

RCT start high, obs. data start

low1. Risk of bias2. Inconsisten

cy3. Indirectnes

s4. Imprecision5. Publication

bias

Gra

de

dow

nG

rad

e

up

1. Large effect

2. Dose response

3. Confounders

Rate quality

of evidence

for each

outcomeSelect

outcomes

Very low

LowModerate

High

Formulate recommendations:

• For or against (direction)• Strong or weak (strength)

By considering: Quality of evidence Balance

benefits/harms Values and

preferences

Revise if necessary by considering:

Resource use (cost)

• “We recommend using…”• “We suggest using…”• “We recommend against using…”• “We suggest against using…”

Outcomes

across

studies

Conclusions

GRADE is gaining acceptance as international standard

Criteria for evidence assessment across questions and outcomes

Criteria for moving from evidence to recommendations

Simple, transparent, systematic Transparency in decision making and

judgments is key