Cullen: Why most published research is wrong

“Why most published

research is wrong.”

Louise Cullen(Clinician researcher)

Disclosure Information

“It is everyone’s responsibility to find out how to

ask questions systematically, find answers from searching the

literature, critically appraise the literature

and apply the results to practice.”

Rinaldo Bellomo

“It is everyone’s responsibility to find out how to

ask questions systematically, find answers from searching the

literature, critically appraise the literature

and apply the results to practice.”

Rinaldo Bellomo

40 ingredients associate with cancer

Most single studies showed implausibly large effects.

The p value

The p value

Observed size of Effect

p=0.01

p=0.01

There is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment.

1000 hypotheses

Replication of studies

Replication of studies

Problems with the study itself.

Wrong question

Wrong Theory

Wrong population studied

2

ACS: Trial and community populations

Circulation. 115(19):2549-69, 2007 May 15.

n=2

Wrong design

• Greater the flexibility in– designs – definitions – outcomes– analytical modes

• Greater the flexibility in– designs – definitions – outcomes– analytical modes

• Hotter a scientific field with more teams involved.

Wrong Endpoints

Ad and high dose Ad

Ca++ in cardiac arrest

COX-2 inhibitors

Milrinone

Methodology

Statistical hypothesis inference testing

Problems with reporting

Interpretation

• “a little significance”

• “a definite trend is evident”

• “a clear tendency”

• “almost achieved significance”

• “a little significance”

• “a definite trend is evident”

• “a clear tendency”

• “almost achieved significance”

The data is practically

meaningless

• “In my experience”• “In case after case”• “In a series of cases”

• “It is generally believed that..”

• “A highly significant area for exploratory study”

• Once• Twice• Three times

• A couple of others think so too

• A totally useless topic in my underpowered study…….

Omitting facts deliberately

….!

Why?

Incentives

Pharma

Pharma

Why?

Ethical practice of researchers

Problems with publishing

Don’t believe in the review process

Journal publishing practices

• 2004 “original articles” in NEJM– 363 tested an established therapy– 146 (40%) reversed that practice– 138 (38%) reaffirmed it

What can you do about it?

Read more than the title!

Reporting Framework

CONSORT (http://bit.ly/14qUNEF)– Standards for reporting of trials

STARD– Standards for the Reporting of diagnostic accuracy

studies

Biases

Be sceptical!

Thank you

@louiseacullen