Topic Refinement

Topic RefinementPrepared for:

The Agency for Healthcare Research and Quality (AHRQ)

Training Modules for Systematic Reviews Methods Guide

www.ahrq.gov

Systematic Review Process Overview

To describe Effective Health Care Program principles

To identify the importance of key questions

To describe the characteristics of a well-constructed key question

To describe the role of key informants in topic refinement

To describe a longitudinal and transparent topic refinement process

Learning Objectives

The Effective Health Care (EHC) Program aims to provide evidence-based information to health care stakeholders.

The work of the EHC Program is intended to be: relevant and timely, objective and scientifically rigorous, and transparent with public participation.

Effective Health Care Program Principles

Whitlock EP, et al. In: Whitlock EP, et al. In: Methods guide for comparative effectiveness reviewsMethods guide for comparative effectiveness reviews . Available at: http://www.effectivehealthcare.ahrq.gov/ . Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction= displayproduct&productid=318.index.cfm/search-for-guides-reviews-and-reports/ ?pageaction= displayproduct&productid=318.

Topic Refinement in the Comparative Effectiveness Review (CER) Process

Develop TopicDevelop Topic

Identify and triage Identify and triage topics based on topics based on appropriateness, appropriateness, importance, importance, desirability of new desirability of new research/duplication, research/duplication, feasibility, and feasibility, and potential impact.potential impact.

Refine TopicRefine Topic

Identify patient Identify patient intervention, intervention, comparator, outcomes, comparator, outcomes, timing, and setting of timing, and setting of each topic.each topic.

Review TopicReview Topic

Prepare topic, search Prepare topic, search for and select studies, for and select studies, abstract data, analyze abstract data, analyze and synthesize data, and synthesize data, and present findings.and present findings.

Engage Stakeholders Engage Stakeholders in Clarifying Areas in Clarifying Areas for Future Researchfor Future Research

Identify future research Identify future research needs and inform real needs and inform real world health care world health care decisions.decisions.

Topics for refinement will be: appropriate, of high priority, feasible, not duplicative, and of

high potential value.

A key feature of topic refinement is the formulation of key questions.

Key questions are objective demonstrations of Effective Health Care Program principles. They should reflect uncertainty stakeholders have

(decisionmakers, clinicians, patients, others).

Topic Refinement

Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/ ?pageaction=displayproduct&productid=318.

Does aliskerin reduce blood pressure in hypertensive patients more than placebo?Is this key question reflecting uncertainties?

No. Aliskerin is approved by the U.S. Food and Drug Administration for hypertension, so it obviously reduces blood pressure more than placebo.

A Question That Does Not Reflect Uncertainty

Does aliskerin reduce blood pressure in hypertensive patients more than other antihypertensives?

Clinicians and decisionmakers would be particularly interested.

Is the antihypertensive effect of aliskerin impacted by ethnicity or genomic parameters?

African Americans and patients with angiotensin-converting enzyme (ACE) gene polymorphisms have less-robust blood pressure reductions with other renin-angiotensin-aldosterone blocking agents.

Patients, clinicians, and decisionmakers would be particularly interested.

Questions That Reflect Uncertainty (I)

Can aliskerin reduce the risk of mortality, myocardial infarction, and stroke in patients with hypertension?

Because thiazide diuretics already reduce blood pressure and reduce final health outcomes, the impact of a new antihypertensive on final health outcomes is important in determining its role in therapy.

Clinicians and decisionmakers are particularly interested.

What is the risk of renal dysfunction, hyperkalemia, cough, and angioedema associated with the use of aliskerin?

Harms need to be considered in order to determine the balance of benefits to harms.

Patients are particularly interested.

Questions That Reflect Uncertainty (II)

Key questions guide the entire systematic review process: Literature search Inclusion and exclusion criteria Types of data extracted Data synthesized and reported

Importance of Getting the Questions Right

Whitlock EP, et al. In: Methods guide for comparative effectiveness reviews. Available at: http://www.effectivehealthcare.ahrq.gov/ index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318.

Population — Who is being evaluated? Intervention — What intervention is being

evaluated? Comparator — What is the intervention being

compared with? Outcome — What are the benefits and harms

being evaluated? Timing — What is the timing of outcomes or

timing of followup? Setting — What are the settings of interest?

Key Questions Derived From Assessing PICOTS


Key questions must be: clear, precise, and relevant to stakeholders.

Key questions should not be posed because: the answers are assumed to be known, or there seems to be adequate literature to evaluate them.

Understand the topic before key questions are posed. Background reading and key informants are critical to

understanding the PICOTS components. Identify appropriate PICOTS components, understand the

topic.

Defining Good Key Questions


PICOTS = population, intervention, comparators, outcomes, timing, and setting

Devise an Analytic Framework*

*See Analytic Frameworks module*See Analytic Frameworks module

Phung OJ, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis [draft report]. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction= displayproduct&productid=391.

Key informants (stakeholders) serve a variety of important functions: They help formulate key questions that address real-world

dilemmas. Key questions should be devised to address important

health care dilemmas, not focus on interests pertinent only to researchers.

They provide context to help discern content areas and applicability.

They ensure transparency in the process.

Roles of Key Informants


Key informants provide context. They explore their beliefs about advantages or

disadvantages of alternative treatments based on pragmatic trials, indirect evidence, understanding of pathophysiology, and understanding of current practice.

They identify important outcomes or information that may affect decisionmaking.

Gaining Context Through Key Informants


Key informants are stakeholders. Clinicians:

clarify (sub)populations to focus on, identify areas in which studies differ that can affect

applicability, and identify what contemporary practice looks like.

Patients: provide insight not usually appreciated by other

stakeholders, and may focus more on quality of life and harms.

Types of Key Informants (I)


Other key stakeholders that can be used as key informants include: employers and business groups, Federal and State partners, health plans, policymakers, and researchers.

Health care industry representatives are key stakeholders but are not included on key informant panels.

Types of Key Informants (II)


After key informants provide insight, a final draft of the topic refinement document is devised and the key question posting document posted for public comment on the Effective Health Care Program Web site.

Public comment allows other stakeholders, including industry representatives, the opportunity to provide input.

Public comment is a critical final check of underlying assumptions about relevance and ensures transparency for all stakeholders.

Public Comment Enhances Inputand Transparency


All public comments that can be answered with the existing Methods Guide are answered generically.

All other public comments are answered individually. Changes to the key questions are made or further

explanations of why certain aspects are not being changed are laid out.

Changes to key questions are incorporated into a draft protocol, which is reviewed at the first Technical Expert Panel meeting.

Revising Key Questions After Public Comment


Step 1: Core leadership team reads topic development document — with title, a preliminary PICOTS description, and a brief review of what systematic reviews and studies exist — and the topic refinement template.

Topic refinement on use of rhGH in cystic fibrosis

Step 2: Core leadership team selects a team from within the Evidence-based Practice Center that has knowledge, interest, and/or contacts within the topic area.

Lead investigator, senior scientist, research fellow identified

Step 3: Selected team performs general review of the topic area, focusing on tertiary texts describing the disease process or interventions of interest and then any practice guidelines.

General review conducted, presented by senior scientist

Topic Refinement Process:UCONN/HH EPC Example (I)

PICOTS = population, intervention, comparators, outcomes, timing, and setting;UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

Step 4: The team reviews the topic development documents and continues more focused background reading.

Step 5: The team proposes the preliminary search strategy. Approved by medical librarian and conducted

Step 6: The citations are reviewed, a body of literature is identified, and data on PICOTS are extracted; no results are reviewed.

Step 7: The team determines the PICOTS and the analytic framework and recruits key informants.

Topic Refinement Process:UCONN/HH EPC Example (II)

PICOTS = population, intervention, comparators, outcomes, timing, and setting;UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

P — Patients with cystic fibrosis

I — Recombinant human growth hormone

C — Placebo, no therapy, another active therapy, or no control therapy

O — Pulmonary variables, anthropometrics, bone variables, IGF variables, hospitalization, bone fracture, health-related quality of life, glucose intolerance, malignancy, and mortality

T — At least 4 weeks of therapy with recombinant human growth hormone

S — Outpatient primary care, endocrine, or cystic fibrosis clinic

Topic Refinement Process:UCONN/HH EPC Example (III)

UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

Topic Refinement Process: UCONN/HH EPC Example (IV)

Phung OJ, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis [draft report]. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction= displayproduct&productid=391.


Step 8: Preliminary key questions are posed. Step 9: List of questions for key informants

is posed. Step 10: Key informants are shown a draft

of the topic refinement document. Step 11: After 2 weeks, the topic refinement

document is discussed with key informants and questions are posed.

Topic Refinement Process:UCONN/HH EPC Example (V)


Questions for key informants Which of the end points available in the trials/studies are the

most important to you as a researcher, clinician, payer, or patient?

Is growth accurately portrayed in this analytic framework as an intermediate outcome, or is it an important health outcome in its own right?

If the data suggest improvements in intermediate outcomes (growth velocity, exercise tolerance, pulmonary function tests, and protein turnover), how comfortable/confident are you that patients will derive the terminal benefits of interest (mortality, infections, fractures, etc.)?

Are there other health outcome benefits that may result from human growth hormone therapy, other than the ones already identified?

Topic Refinement Process:UCONN/HH EPC Example (VI)


Questions for Key Informants How different is a study including people up to 23 years of

age from a study limited to 18-year-olds? Consider this given the inverse linear relationship between human growth hormone efficacy and age, as well as the delayed growth and maturation inherent in this population.

How readily do you feel we can combine certain end points (e.g., lean body mass measured by different methods [DEXA, skin folds, BIA] in same units) or amino acid catabolism measures (leucine vs. glutamine)?

Topic Refinement Process:UCONN/HH EPC Example (VII)

BIA = bioelectric impedance analysis; DEXA = dual energy x-ray absorptometry;UCONN/HH EPC = University of Connecticut/Hartford Hospital Evidence-based Practice Center

Questions for Key Informants: In trials/studies that report multiple similar end points (like

pulmonary function tests), which surrogate end points would be optimal (i.e., FEV1 percent vs. FEV1 absolute value)?

Given the paucity of harms data, could harms data from patients without cystic fibrosis be used to inform the potential risks associated with human growth hormone therapy (i.e., how similar are patients with hypogonadism to those with cystic fibrosis and poor growth progression)? In particular, are you wary of using human growth

hormone because of its risk of neoplasms?

Topic Refinement Process:UCONN/HH EPC Example (VIII)

FEV1 = forced expiratory volume in 1 second; UCONN/HH EPC =University of Connecticut/Hartford Hospital Evidence-based Practice Center

Step 12: Based on feedback, the revised topic refinement and key question posting document is generated.

Step 13: The topic refinement and key question posting document is sent to AHRQ.

Step 14: After editorial review, the key question posting document is posted for public comment on the Effective Health Care Program Web site.

Topic Refinement Process:UCONN/HH EPC Example (IX)


Step 15: Public comments are sent to the Evidence-based Practice Center, and revisions are made in response to the comments.

Step 16: Changes to key questions are incorporated into a draft protocol that is reviewed at the first Technical Expert Panel meeting.

Topic Refinement Process:UCONN/HH EPC Example (X)


KQ 1: In patients with cystic fibrosis, does treatment with rhGH as an adjuvant to usual care improve intermediate outcomes, including pulmonary function (FEV percent and FEV1 absolute value), growth (height, weight, lean body mass, protein turnover), exercise tolerance, and bone mineralization, compared with usual care alone?

KQ 2: In patients with cystic fibrosis, does treatment with rhGH as an adjuvant to usual care improve health outcomes, including frequency of required intravenous antibiotic treatments, frequency of hospitalization, quality of life, bone fracture or development of osteoporosis/osteopenia, or mortality, compared with usual care alone?

KQ 3: In patients with cystic fibrosis, what is the strength of evidence that intermediate outcomes of pulmonary function, growth, and bone mineralization are associated with improvements in health outcomes of quality of life, bone fracture or development of osteoporosis/osteopenia, or mortality?

KQ 4: In patients with cystic fibrosis, what is the frequency of nonmalignant serious adverse effects resulting from treatment with rhGH? Adverse effects of interest include, but are not limited to: glucose intolerance, diabetes, and hypoglycemia.

Key Questions (I)

KQ 5: What is the risk of malignancy associated with rhGH use as determined by (a) markers of cancer risk with rhGH (IGF-I increases over 100 ng/ml or IGFBP-3 decreases over 1,000 ng/ml) from studies of rhGH in people with CF [cystic fibrosis], and (b) assessment of evidence on cancer incidence from non-CF patients receiving modest doses of rhGH (0.2 mg/kg per week to 0.6 mg/kg per week) for disorders such as growth hormone deficiency and idiopathic short stature?

KQ 6: In patients with CF, how are efficacy, effectiveness, safety, or adverse events impacted by rhGH dose, therapy duration, baseline nutritional status, and concurrent medical therapies?

KQ 7: In patients with CF, how do the efficacy, effectiveness, safety, or adverse events of treatment with rhGH differ between subgroups of patients? Subgroup characteristics of interest include, but are not limited to, age (prepubertal, pubertal, postpubertal); gender; baseline clinical status (height, weight, lean body mass, pulmonary function, exercise tolerance, nutritional status); and/or the nature, extent, and effectiveness of prior treatment.

Key Questions (II)

The topic refinement process is devised to ensure quality, clinical relevance, and transparency.

Understanding the topic area is important to describing the PICOTS and devising the analytic framework.

Studies are evaluated in this phase to review methods and demographics, not results.

Key informants provide context and ensure relevance and transparency.

Key informants cannot be limited to researchers. Public comment is a final verification of relevance,

and ensures transparency and a broad range of input.

Key Messages

Phung OJ, Coleman CI, Baker EL, et al. Effectiveness of Recombinant Human Growth Hormone (rhGH) in the Treatment of Patients With Cystic Fibrosis, Draft Comparative Effectiveness Review (Prepared by University of Connecticut/Hartford Hospital Evidence-based Practice Center under Contract No. 290-2007-10067-I). Rockville, MD: Agency for Healthcare Research and Quality, Posted March 2010. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=391.

Whitlock EP, Lopez SA, Chang S, et al. Identifying, selecting, and refining topics. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; posted April 2009. Available at: http://www.effectivehealthcare.ahrq. gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction= displayproduct&productid=318.

References

This presentation was prepared by C. Michael White, Pharm.D., FCP, FCCP, a member of the University of Connecticut/Hartford Hospital Evidence-based Practice Center.

The module is based on chapter 2 in version 1.0 of the Methods Reference Guide for Effectiveness and Comparative Effectiveness Reviews (available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf).

Author

Topic Refinement

Health & Medicine

importance of key questions

formulation of key questions

review topic

topic refinement whitlock

health care stakeholders

systematic reviews methods

role of key informants

questions right whitlock