Searching Grey Literature: Is the Effort Worth the Yield? AHRQ Annual Meeting Track C, Session 105 Methods for Synthesizing Evidence Marian McDonagh, PharmD.

Searching Grey Literature: Is the Effort Worth the Yield?AHRQ Annual Meeting Track C, Session 105 Methods for Synthesizing Evidence

Marian McDonagh, PharmDEvidence-based Practice Center

The Problem:

• Searching for and including grey literature adds substantial workload to a systematic review.

• The benefit is unclear in terms of value for time spent

• Information from grey literature may be limited and risk of bias cannot be determined.

2

Session Purpose and Goals

• This session will explore –The methodological argument in

favor of searching for grey literature–Different approaches to searching

for grey literature with highest yield –The usefulness of including grey

literature based on real-world examples

3

Discussion Leaders

• Dr. David Moher– Ottawa Hospital Research Institute – University of Ottawa– Ottawa Evidence-based Practice Center

• Rose Relevo, MLIS, MS– Oregon Evidence-based Practice Center– Oregon Health & Science University

• Marian McDonagh, PharmD– Oregon Evidence-based Practice Center– Oregon Health & Science University

Usefulness of Grey Literature

• What have we discussed so far today:• Evidence that failure to include grey literature

can lead to biased result (Dr. Moher)– Methods to identify grey literature for comparative

effectiveness reviews (Ms. Relevo)

• Interactive Session: What’s it like in practice? – Examples from the Drug Effectiveness Review

Project use of FDA materials as grey literature

Drug Effectiveness Review Project

• DERP is a collaboration of primarily state Medicaid agencies that have joined together to commission systematic reviews of the comparative effectiveness and harms of drugs in many widely used drug classes, and to apply the findings to inform public policy in local settings.

• Project began in 2003– OHSU Center for Evidence-based Policy coordinates– EPCs at Oregon and University of North Carolina

produce systematic reviews for the project.

6

DERP I (15)2003-2006

DERP II (17)2006-2009

DERP III (12)2009 - 2012

AlaskaArkansasCADTH*California HCFIdahoKansasMichiganMinnesotaMissouriMontanaNorth CarolinaOregonWashingtonWisconsinWyoming

AlaskaArkansasCaliforniaCalifornia HCFCADTH*IdahoKansasMichiganMinnesotaMissouriMontanaNew YorkNorth CarolinaOregonWashingtonWisconsinWyoming

ArkansasCADTH*ColoradoIdahoMarylandMissouriMontanaNew YorkOregonWashingtonWisconsinWyoming

*Canadian Agency for Drugs and Technologies in Health

DERP Organizations

114 Reports Completed to date

Proton Pump Inhibitors

Long-acting Opioids

Statins

Non-steroidal Anti-Inflammatory Drugs (NSAIDs)

Estrogens

Triptans

Skeletal Muscle Relaxants

Oral Hypoglycemics

Drugs for Overactive Bladder

ACE Inhibitors

Beta Blockers

Calcium Channel Blockers

Angiotensin II Receptor Antagonists

2nd Generation Antidepressants

Drugs for Constipation

Direct Renin Inhibitors/ACEI/ AIIRA

Fibromyalgia, Drugs to treat

Drugs for Neuropathic Pain

Antiepileptic Drugs in Bipolar Disorder/ Pain

2nd Generation Antihistamines

Atypical Antipsychotics

Inhaled Corticosteroids

ADHD and ADD, Drugs to treat

Alzheimer's, Drugs to treat

Antiplatelet Drugs

Thiazolidinediones (glitazones)

Drugs for Hepatitis C

Newer Drugs for Insomnia

Targeted Immune Modulators

Beta Agonists

Newer Anti-emetics

Drugs for Multiple Sclerosis

Combination Products

Controller Drugs for Asthma

Newer Drugs for Diabetes

Topical Calcineurin Inhibitors

DERP Methods in Using FDA Documents

• Systematic review methodology for DERP includes searching FDA web site– FDA Medical and Statistical reviews of NDAs

• Documents available on FDA web site > 1997• Approved drugs, newly approved for updates• Include studies if enough information to rate quality

• What we are looking for:– Unique unpublished studies– Unpublished data related to published studies

Workload?

• Number of documents reviewed varies• Range 0 to 9 FDA documents per report (mean 1.5)

• Document Size and Organization Issues• Several hundred pages each• Organization varies• Historically pdfs of scanned originals

• Matching published and unpublished studies• Review after electronic searching and selection

• Dual review: eligibility, quality, abstraction

DERP Studying Experience with FDA Documents

• We are analyzing DERP reports for• Frequency of information found in FDA documents

being included in a report• Value of information from FDA documents included• Value is being assessed by assigning an impact

category to each instance, e.g. fills a gap in evidence

• 175 FDA documents eligible for review in114 reports over 9 years– 48 instances of evidence from FDA documents

included (27%)

Example 1: Unique unpublished study

• Atypical Antipsychotic Drugs (2004)– Head to head comparisons of primary

relevance to review– No eligible published studies of aripiprazole

(Abilify®) identified through usual searches and request for manufacturer information

– Abilify® FDA document found• > 200 hundred pages, Examined 5 trials• Identified one trial head to head trial (N = 255)

Your thoughts?

If you came across this evidence what would do with it?

1.Nothing. Don’t use grey literature.

2.Include in report qualitatively only

3.Include in report qualitatively, but influences publication bias assessment

4.Fully include, incorporate into meta-analysis, adjust strength of evidence

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1.Yes

2.No

3.Unsure

Your Thoughts?

Do you believe the added information was valuable to the review’s conclusions?

1.Yes

2.No

3.Unsure

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1.Yes

2.No

3.Unsure

Discussion

• Panelists Thoughts?• What we did in DERP

– Included study in report; changed conclusions only in a minor way• Secondary outcome measure used (cognitive

function)• Aripiprazole superior to olanzapine on some, but

not all measures

• Further discussion?

Example 2: Unpublished data

• Drugs for Insomnia, Update 2 (2008)• Zolpidem ER (Ambien CR®) new

– 3 published placebo-controlled trials identified– 5 placebo-controlled trials in FDA documents

• Matched 2 of the published trials

• FDA docs: Primary outcome at 8 hours – Ambien superior to placebo up to 6 hours, not

different at hours 7 and 8– Publication: reports only 6-hour results

Your thoughts?

If you came across this evidence what would do with it?

1.Nothing. Don’t use grey literature.

2.Include in report qualitatively only

3.Include in report qualitatively, but influences publication bias assessment

4.Fully include, incorporate into meta-analysis, adjust strength of evidence

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1.Yes

2.No

3.Unsure

Your Thoughts?

Do you believe the added information was valuable to the review’s conclusions?

1.Yes

2.No

3.Unsure

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1.Yes

2.No

3.Unsure

Discussion

• Panelists Thoughts?• What we did in DERP

– Included results from FDA docs – changed conclusions of report on Ambien CR®

• Further discussion?

Example 3: Unique unpublished study

• Proton Pump Inhibitors, Update 2 (2004)• Esomeprazole (Nexium®) vs omeprazole

– FDA documents: 4 trials submitted prior to the 2001 approval of Nexium®

– 2 published studies identified • Both found esomeprazole superior to omeprazole

– 2 unpublished studies (in FDA docs) found no statistically significant difference between the drugs

Your thoughts?

If you came across this evidence what would do with it?

1.Nothing. Don’t use grey literature.

2.Include in report qualitatively only

3.Include in report qualitatively, but influences publication bias assessment

4.Fully include, incorporate into meta-analysis, adjust strength of evidence

Your Thoughts?

If you had included this information, how do you think it would have changed the SOE?

1.Yes

2.No

3.Unsure

Your Thoughts?

Do you believe the added information was valuable to the review’s conclusions?

1.Yes

2.No

3.Unsure

Your thoughts?

Do you think that the time and resources spent to obtain this information was worthwhile?

1.Yes

2.No

3.Unsure

Discussion

• Panelists Thoughts?• What we did in DERP

– Included study in report only qualitatively– Inadequate information in FDA docs to

evaluate quality– Subsequent update: obtained data from

manufacturer and incorporated into meta-analysis

• Further discussion?

Wrap up

• The Problem Revisited– Searching for and including grey literature

adds substantial workload to a systematic review.

– The benefit is unclear in terms of value for time spent

– Information from grey literature may be limited and risk of bias cannot be determined.