Chapter019

Copyright © 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Chapter 19

Systematic Reviews: Meta-Analysis and Metasynthesis


Research Integration and EBP

• The systematic and rigorous integration and synthesis of evidence is a cornerstone of EBP

• Impossible to develop “best practice” guidelines, protocols, and procedures without organizing and evaluating research evidence through a systematic review


Research Integration and Synthesis• Forms of systematic reviews:

– Narrative, qualitative integration (traditional review of quantitative or qualitative results)

– Meta-analysis (statistical integration of results)

– Metasynthesis (theoretical integration and interpretation of qualitative findings)


Meta-Analysis: Advantages

• Objectivity—statistical integration eliminates bias in drawing conclusions when results in different studies are at odds

• Increased power—reduces risk of Type II error compared to single study

• Increased precision—results in smaller confidence intervals than single studies


Meta-Analysis: Criteria

• Research question or hypothesis should be essentially identical across studies.

– The “fruit” problem—don’t combine apples and oranges!

• Must be a sufficient knowledge base—must be enough studies of acceptable quality

• Results can be varied but not totally at odds.


Question

Is the following statement True or False?

• An advantage of meta-analysis is that it reduces the risk of a Type I error as compared to a single study.


Answer

• False

– Meta-analysis increases power and thus reduces the risk of a Type II error when compared to a single study.


Major Steps in a Meta-Analysis

• Delineate research question or hypothesis to be tested.

• Identify sampling criteria for studies to be included.

• Develop and implement a search strategy.

• Locate and screen sample of studies meeting the criteria.


Major Steps in a Meta-Analysis (cont.)

• Appraise the quality of the study evidence.

• Extract and record data from reports.

• Formulate an analytic plan (i.e., make analytic decisions).

• Analyze data according to plan.

• Write a systematic review.


Search Strategy

• Identify electronic databases to use.

• Identify additional search strategies (e.g., ancestry approach).

• Decide whether or not to pursue the gray literature (unpublished reports).

• Identify keywords for the search:

– Think creatively and broadly.


Evaluating Study Quality

• Meta-analysts must make decisions about handling study quality.

• Approaches:

– Omit low-quality studies (e.g., in intervention studies, non-RCTs).

– Give more weight to high-quality studies.

– Analyze low- and high-quality studies to see if effects differ (sensitivity analyses).


Evaluating Study Quality (cont.)

• Evaluations of study quality can use:

– A scale approach (e.g., use a formal instrument to “score” overall quality)

– A component approach (code whether certain methodologic features were present or not, e.g., randomization, blinding, low attrition)


Question

After identifying the research question to be tested for a meta-analysis, which would the researcher complete next?

a. Develop a search strategy

b. Locate sample of studies

c. Identify sampling criteria

d. Extract data from reports


Answer

c. Identify sampling criteria

• Once the research question has been delineated, the next step is to identify sampling criteria for studies to be included. Then the researcher develops and implements a search strategy, locating and screening the sample of studies that meet the criteria. Next the researcher appraises the quality of the study evidence and extracts and records data from the reports.


Analytic Decisions in Meta-Analysis

• Decisions include:

– What effect size index will be used?

– How will heterogeneity be assessed?

– Which analytic model will be used?

– Will there be subgroup (moderator) analyses?

– How will quality be addressed?

– Will publication bias be assessed?


Effect Size Indexes

• A central feature of meta-analysis is the calculation of an effect size index for each study that encapsulates the study results.

• An ES index is computed for each study and then combined and averaged (often weighted for sample size).

• Several different effect size (ES) indexes can be used.


Effect Size Indexes (cont.)

• Major effect size indexes:

– d: the standardized difference between 2 groups (e.g., Es vs. Cs) on an outcome for which a mean can be calculated (e.g., BMI)

– Odds Ratio (OR): relative odds for two groups on a dichotomous outcome (e.g., smoke/not smoke)

– r: correlation between 2 continuous variables (e.g., age and depression)


Heterogeneity• Results (effects) inevitably vary from one study to

the next.

• Major question: Is heterogeneity just random fluctuations?

– If “yes,” then a fixed effects model of analysis can be used.

– If “no,” then a random effects model should be used.

• Heterogeneity can be formally tested but also can be assessed visually via a forest plot.


Heterogeneity (cont.)• Factors influencing variation in effects is

usually explored via subgroup analysis (moderator analysis).

• Do variations relate to:

– Participant characteristics (e.g., men vs. women)?

– Methods (e.g., RCTs vs. quasi-experiments)?

– Intervention characteristics (e.g., 3-week vs. 6-week intervention)?


Question


• A key component of meta-analysis is the calculation of an effect size index.


Answer

• True

– An effect size index is a central feature of meta-analysis. It is computed for each study and then combined and averaged.


Publication Bias• Nonpublished studies are more likely to have

no effects or weak effects than published ones.

– So…excluding them could result in overestimating effects.

• One approach:

– Compute a fail-safe number to see how many studies with 0 effect would be needed to change conclusions from significant to nonsignificant.


Metasynthesis

• One definition: The bringing together and breaking down of findings, examining them, discovering essential features, and combining phenomena into a transformed whole

• Integrations that are more than the sum of the parts—novel interpretations of integrated findings


Metasynthesis: Some Ongoing Debates

• Whether to exclude low-quality studies

• Whether to integrate studies based in multiple qualitative traditions

• Various typologies and approaches; differing terminology


Metasynthesis: Steps• Similar to meta-analysis in many ways

– Formulate question

– Decide selection criteria, search strategy

– Search for and locate studies

– Extract data for analysis

– Formulate and implement an analysis approach

– Integrate, interpret, write up results


Metasynthesis Approaches

• Noblit and Hare (developed an approach for a meta-ethnography)

– Suggest a 7-phase approach

– Involves “translating” findings from qualitative studies into one another

– An “adequate translation maintains the central metaphors and/or concepts of each account”

– Final step is synthesizing the translations


Metasynthesis Approaches (cont.)

• Paterson and colleagues’ approach involves three components:

– Meta-data analysis (analyzing and integrating the study findings)

– Meta-method (analyzing the methods and rigor of studies in the analysis)

– Meta-theory (analysis of the studies’ theoretical underpinnings)


Metasynthesis Approaches (cont.)

• Sandelowski and Barrosa’s approach distinguishes studies that are summaries (no conceptual reframing) and syntheses (studies involving interpretation and metaphorical reframing).

• Both summaries and syntheses can be used in a meta-summary, which can lay a foundation for a metasynthesis.


Question


• According to Sandelowski and Barrosa, a meta-summary lays the foundation for meta-analysis.


Answer

• False

– A meta-summary, as described by Sandelowski and Barrosa, lays the foundation for a metasynthesis.


Meta-Summaries

• Involve making an inventory of findings and can be aided by computing manifest effect sizes (effect sizes calculated from the manifest content in the studies in the review).

• Two types:

– Frequency effect size

– Intensity effect size


Effect Sizes in Meta-Summaries• Frequency effect size

– Count the total number of findings across all studies in the review (specific themes or categories).

– Compute prevalence of each theme across all reports (e.g., the #1 theme was present in 75% of reports).

• Intensity effect size

– For each report, compute how many of the total themes are included (e.g., report 1 had 60% of all themes identified).


Sandelowski and Barroso’s Metasynthesis

• Can build on a meta-summary

• But can only be done with studies that are syntheses (not summaries), because the purpose is to offer novel interpretations of interpretive findings—not just summaries of findings


End of Presentation

Chapter019

Documents

false metaanalysis

advantage of metaanalysis

highquality studies

study quality metaanalysts

different studies

single study

intervention studies

lowquality studies