1 Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings? Erik Fernandez y Garcia, MD MPH University of California, Davis Co-authors: Hien Nguyen, MD (UCD); Naihua Duan, PhD (Columbia University); Nicole Bloser Gabler, MHA MPH (UCD); Diana Liao MPH(UCLA); Richard L. Kravitz, MD MSPH (UCD)
22
Embed
Erik Fernandez y Garcia , MD MPH University of California, Davis
Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their Own Findings?. Erik Fernandez y Garcia , MD MPH University of California, Davis - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Subgroup Reporting in the General Medical Literature: Do Investigators Misinterpret Their
Own Findings?
Erik Fernandez y Garcia, MD MPHUniversity of California, Davis
– Odd months in 1994, 1999, 2004– Initial search: N = 4,863 articles – After additional random sampling and
exclusions, N = 319 clinical trials– Final sample: 87 of 319 trials (27%) reporting
test for HTE
7
Covariates Examined in HTE analyses
Prespecified Not Prespecified
All covariates Some
Substantive Statistical
Number of Covariates with Rationale
None
Types of Reasons
Coding of Covariates
8
Defining Clinicostatistical Divergence
Clinicostatistical Divergence: Clinically meaningful and statistically significant differences between subgroup effects and average effect (coded as “none” “weak” “moderate” “strong”)
– Clinical Divergence (CD): Was the ratio measure of effect in any subgroup at least 25% greater or smaller than in the sample as a whole?
– Statistical Significance (SS): Was a test for interaction associated with a p value of less than or equal to 0.10?
9
Coding Clinicostatistical Divergence
* Denotes absence of data
Strength of Evidence
CD
SS
Prespecified Covariate?
Strong Yes Yes Yes Moderate Yes Yes No Weak Yes No Weak Yes * Weak No Yes Weak * Yes None No No None * No None No * Unable to Classify * *
10
Coding Authors’ Interpretations
• Evidence for HTE sufficient to support different treatment recommendations in one or more subgroups
• Evidence for HTE insufficient to support different treatment recommendations but sufficient to warrant further systematic research
• Evidence for HTE was possibly present but insufficient to warrant further research
• Definite evidence against HTE
• No interpretation of HTE results
11
87 RCTs
Prespecified Not Prespecified
All covariates Some
Substantive Statistical
Trials by Number of Covariates with Rationale
None
Types of Reasons
RESULTS
53 (61%) 34 (39%)
17 (32%) 12 (23%) 24 (45%)
22 (76%) 7 (24%)
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
29/87 = 33%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
2 5 1 2
Warrants Further Research
6 9 5 1
Possibly Present
0 0 0 1
Evidence Against
2 8 2 2
No Interpretation
6 3 3 0
Total 16 25 11 6 58
17/58 = 29%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
25/87 = 29%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
Total 62
31/62 = 50%10/62 = 16%31/62 = 50%
Overstated = 27% Understated = 9%
Strength of Evidence Author’s Interpretation Unable None Weak Moderate Strong Total Supports Differential Treatment
0 2 5 1 2 10
Warrants Further Research
10 6 9 5 1 31
Possibly Present
0 0 0 0 1 1
Evidence Against
6 2 8 2 2 20
No Interpretation
13 6 3 3 0 25
Total 29 16 25 11 6 87
18
LIMITATIONS
• Limited number of journals, years, trials reviewed
• Data potentially incomplete– HTE analyses performed but not published
– HTE analyses performed and published in secondary journals
19
CONCLUSIONS
• Analysis and reporting of HTE incomplete
– Prespecification inconsistent
– Rationale incomplete
– Effect measures and p-values (or CIs) incompletely reported
• When reported, objective evidence for clinicostatistical divergence found in approximately 1/3 of trials
• Authors frequently misinterpret own findings (in both directions)
20
IMPLICATIONS• Researchers:
– Ensure that SGA are prespecified with a priori rationales for covariate inclusion, or clearly labeled as exploratory
– Include a statistical test for interaction or heterogeneity in the analyses
– Report all results from SGA (including p values for HTE tests and effect measures with confidence intervals), even if not significant
21
IMPLICATIONS
• Journal Editors:– Ensure authors report SGA-associated data
when SGA is performed – Ensure that authors’ discussion includes
interpretation of SGA performed and limitations of such analyses
• Readers/Clinicians:– Weigh the authors’ interpretations and
recommendations in light of the objective evidence presented prior to changing practice or implementing recommendations