Top Banner
Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov
11

Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Apr 01, 2015

Download

Documents

Stacy Horsey
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Study Size Planning for Observational Comparative

Effectiveness Research

Prepared for:Agency for Healthcare Research and Quality (AHRQ)

www.ahrq.gov

Page 2: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

This presentation will: Describe all relevant assumptions and decisions Specify the type of hypothesis, the clinically

important inferiority margin or minimum clinically important excess/difference, and the level for the confidence interval

Specify the statistical software and command or the formula to calculate the expected confidence interval

Specify the expected precision (or statistical power) for any subgroup analyses

Specify the expected precision (or statistical power) as sensitivity analyses in special situations

Outline of Material

Page 3: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Study feasibility relies on whether the projected number of accrued patients is adequate to address the scientific aims of the study.

Many journal editorial boards endorse reporting of study size rationale. However, this rationale is often missing from study

protocols and proposals. Interpreting study findings in terms of statistical

significance in relation to the null hypothesis implies a prespecified hypothesis and adequate statistical power.

Without the context of a numeric rationale for the study size, readers may misinterpret the results.

Introduction

Page 4: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Reporting on study size rationale in the study protocol is often required by institutional review boards before data collection can begin.

The rationale for study size depends on calculations of the study size needed to achieve a specified level of statistical power.

Statistical power is defined as the probability of rejecting the null hypothesis when an alternative hypothesis is true.

Software packages and online tools can assist with these calculations.

Study Size and Power Calculations in Randomized Controlled Trials (1 of 3)

Page 5: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Specify the clinically meaningful or minimum detectable difference. Identify the size of the smallest potential treatment

effect that would be of clinical relevance. Calculate the study size, assuming the value

represents the true treatment effect. Specify a measure of data variability.

For continuous outcomes, make assumptions about the standard deviation.

For occurrence of event outcomes (e.g., death), estimation of the assumed event rate in the control group is necessary.

Study Size and Power Calculations in Randomized Controlled Trials (2 of 3)

Page 6: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Needed study size depends on the chosen type 1 error rate (α) and required statistical power. Use a conventional statistical significance cutoff of α =

0.05 and a standard required power of 80 percent. Consider potential reductions in the number of recruited

patients available for analysis.

Study Size and Power Calculations in Randomized Controlled Trials (3 of 3)

ScenarioEffect of Interest

Therapy 1 Risk

Therapy 2 Risk

Desired

Power

Needed Study Size

Needed Recruitme

nt

1 0.75 0.020 0.015 80% 10,795 13,494

2 0.75 0.100 0.075 80% 2,005 2,507

3 0.50 0.100 0.050 80% 435 544

4 0.50 0.100 0.050 90% 592 728

An example of adequately reported consideration of study size under several potential scenarios that vary the baseline risk of the outcome, the minimum clinically relevant treatment effect, and the required power.

Page 7: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Sample size and power calculations in the context of randomized controlled trials are relevant for observational studies, but their application may differ. Strengthening the Reporting of Observational Studies

in Epidemiology (STROBE) guidelines Funding agencies often ask for statistical power

calculations, while journal editors ask for confidence intervals.

Considerations for Observational Comparative Effectiveness Research Study Size Planning

Page 8: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Confounding bias, measurement error, and other biases should concern investigators more than the expected precision when they consider the feasibility of an observational comparative effectiveness study.

Controlling for confounding can also reduce the precision of estimated effects (often seen in studies with propensity score matching).

Retrospective studies often suffer from a higher frequency of missing data, which can limit precision and power.

Considerations That Differ FromNonrandomized Studies

Page 9: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

To ensure adequate study size and appropriate interpretation of results, provide a rationale for study size during the planning and reporting stages.

All definitions and assumptions should be specified, including primary study outcome, clinically important minimum effect size, variability measure, and type I and type II error rates.

Consider loss to followup, reductions due to statistical methods to control for confounding, and missing data to ensure the sample size is adequate to detect clinically meaningful differences.

Conclusions

Page 10: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Summary Checklist (1 of 2)

Guidance Key Considerations

Describe all relevant assumptions and decisions.

• Report the primary outcome on which the study size or power estimate is based.

• Report the clinically important minimum effect size (e.g., hazard ratio ≥1.20).

• Report the type I error level.• Report the statistical power or type II error level (for study size

calculations) or the assumed sample size (for power calculations).• Report the details of the sample size formulas and calculations

including correction for loss to followup, treatment discontinuation, and other forms of censoring. Report the expected absolute risk or rate for the reference or control cohort, including the expected number of events.

Specify the type of hypothesis, the clinically important inferiority margin or minimum clinically important excess/difference, and the level of confidence for the interval (e.g., 95%).

• Types of hypotheses include equivalence, noninferiority, and inferiority.

Page 11: Study Size Planning for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) .

Summary Checklist (2 of 2)

Guidance Key Considerations

Specify the statistical software and command or the formula to calculate the expected confidence interval.

• Examples include Stata, Confidence Interval Analysis, and Power Analysis and Sample Size (PASS).

Specify the expected precision (or statistical power) for any planned subgroup analyses.

Specify the expected precision (or statistical power) as sensitivity analyses in special situations.

Special situations include:• The investigators anticipate strong confounding that will

eliminate many patients from the analysis (e.g., when matching or trimming on propensity scores).

• The investigators anticipate a high frequency of missing data that cannot (or will not) be imputed, which would eliminate many patients from the analysis.