Re-imagining failure when conceptualising research ... · vid Moher,, Davina Ghersi, Douglas G Altman, Elaine Beller, Elina Hemminki, ElizabethWager, Fujian Song, Harlan M Krumholz,

ily Sena, Gillian Currie, Hanna Vesterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca, Zsannet Bahor, Theo Hirst, Kim Wever, Hugo Pedder, Katerinaiacopoulou, Julija Baginskaite, Ye Ru, Stelios Serghiou, Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, Andrew Thomson, Aparna Poltururah MaCann, Gillian Mead, Joanna Wardlaw, Rustam Salman, Joseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle, Andrew Rice, Rosie Morelandhalie Percie du Sert, Paul Garner, Lauralyn McIntyre, Gregers Wegener, Lindsay Thomson, David Howells, Ana Antonic, Tori O’Collins, Uli Dirnagl, H Barder Worp, Philip Bath, Mharie McRae, Stuart Allan, Ian Marshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis Panagiotou, John Ioannidis, Pete

chelor, David Howells, Sanne Jansen of Lorkeers, Geoff Donnan, Peter Sandercock, A Metin Gülmezoglu, Andrew Vickers, An-Wen Chan, Ben Djulbegovicvid Moher,, Davina Ghersi, Douglas G Altman, Elaine Beller, Elina Hemminki, Elizabeth Wager, Fujian Song, Harlan M Krumholz, Iain Chalmers, Ian Robertsbelle Boutron, Janet Wisely, Jonathan Grant, Jonathan Kagan, Julian Savulescu, Kay Dickersin, Kenneth F Schulz, Mark A Hlatky, Michael B Bracken, Mikerke, Muin J Khoury, Patrick Bossuyt, Paul Glasziou, Peter C Gøtzsche, Robert S Phillips, Robert Tibshirani, Sander Greenland, Sandy Oliver, Silviorattini, Steven Julious, Susan Michie, Tom Jefferson, Emily Sena, Gillian Currie, Hanna Vesterinen, Kieren Egan, Nicki Sherratt, Cristina Fonseca, Zsannehor, Theo Hirst, Kim Wever, Hugo Pedder, Katerina Kyriacopoulou, Julija Baginskaite, Ye Ru, Stelios Serghiou, Aaron McLean, Catherine Dick, Traceyodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian Mead, Joanna Wardlaw, Rustam Salman, Joseph Frantzias, Robin Grantul Brennan, Ian Whittle, Andrew Rice, Rosie Moreland, Nathalie Percie du Sert, Paul Garner, Lauralyn McIntyre, Gregers Wegener, Lindsay Thomson, Davidwells, Ana Antonic, Tori O’Collins, Uli Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae, Stuart Allan, Ian Marshall, Xenios Mildonis, Konstantinosidis, Orestis Panagiotou, John Ioannidis, Peter Batchelor, David Howells, Sanne Jansen of Lorkeers, Geoff Donnan, Peter Sandercock, A Metin Gülmezogludrew Vickers, An-Wen Chan, Ben Djulbegovic, David Moher,, Davina Ghersi, Douglas G Altman, Elaine Beller, Elina Hemminki, Elizabeth Wager, Fujianng, Harlan M Krumholz, Iain Chalmers, Ian Roberts, Isabelle Boutron, Janet Wisely, Jonathan Grant, Jonathan Kagan, Julian Savulescu, Kay Dickersinnneth F Schulz, Mark A Hlatky, Michael B Bracken, Mike Clarke, Muin J Khoury, Patrick Bossuyt, Paul Glasziou, Peter C Gøtzsche, Robert S Phillips, Robershirani, Sander Greenland, Sandy Oliver, Silvio Garattini, Steven Julious, Susan Michie, Tom Jefferson, Emily Sena, Gillian Currie, Hanna Vesterinen, Kierenan, Nicki Sherratt, Cristina Fonseca, Zsannet Bahor, Theo Hirst, Kim Wever, Hugo Pedder, Katerina Kyriacopoulou, Julija Baginskaite, Ye Ru, Steliosrghiou, Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian Mead, Joanna Wardlawstam Salman, Joseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle, Andrew Rice, Rosie Moreland, Nathalie Percie du Sert, Paul Garner, Lauralynntyre, Gregers Wegener, Lindsay Thomson, David Howells, Ana Antonic, Tori O’Collins, Uli Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae, Stuar

an, Ian Marshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis Panagiotou, John Ioannidis, Peter Batchelor, David Howells, Sanne Jansen of Lorkeersoff Donnan, Peter Sandercock, A Metin Gülmezoglu, Andrew Vickers, An-Wen Chan, Ben Djulbegovic, David Moher,, Davina Ghersi, Douglas G Altmanine Beller, Elina Hemminki, Elizabeth Wager, Fujian Song, Harlan M Krumholz, Iain Chalmers, Ian Roberts, Isabelle Boutron, Janet Wisely, Jonathan Grantathan Kagan, Julian Savulescu, Kay Dickersin, Kenneth F Schulz, Mark A Hlatky, Michael B Bracken, Mike Clarke, Muin J Khoury, Patrick Bossuyt, Pausziou, Peter C Gøtzsche, Robert S Phillips, Robert Tibshirani, Sander Greenland, Sandy Oliver, Silvio Garattini, Steven Julious, Susan Michie, Tomferson Aaron McLean, Catherine Dick, Tracey Woodruff, Patrice Sutton, Andrew Thomson, Aparna Polturu, Sarah MaCann, Gillian Mead, Joanna Wardlawstam Salman, Joseph Frantzias, Robin Grant, Paul Brennan, Ian Whittle, Andrew Rice, Rosie Moreland, Nathalie Percie du Sert, Paul Garner, Lauralynntyre, Gregers Wegener, Lindsay Thomson, David Howells, Ana Antonic, Tori O’Collins, Uli Dirnagl, H Bart van der Worp, Philip Bath, Mharie McRae, Stuar

an, Ian Marshall, Xenios Mildonis, Konstantinos Tsilidis, Orestis Panagiotou, John Ioannidis, Peter Batchelor, David Howells, Sanne Jansen of Lorkeersoff Donnan, Peter Sandercock, A Metin Gülmezoglu, Andrew Vickers, An-Wen Chan, Ben Djulbegovic, David Moher,, Davina Ghersi, Douglas G AltmanCAMARADES: Bringing evidence to translational medicine

Re-imagining failure when conceptualising research:

translational researchMalcolm Macleod

Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies

andUniversity of Edinburgh

Presenter

Presentation Notes

25 minutes : failure as a lever for research improvement Biomedical research: Is failed replication always a bad thing? Failed replication in biomedical research may occur if the originator study was falsely positive (by chance, or because the experimental design placed the study at risk of bias); if our understanding of the literature is polluted by publication bias; or in the presence of some unknown (latent) independent variable which influences the phenomenon under study. By definition, this latter explanation promises the discovery of a previously unknown facet of the process being studied. Approaches to replication studies might focus on the parallel tasks of increasing the probability that published research is true; and then exploring the possible causes for unexplained conflicting results.

CAMARADES: Bringing evidence to translational medicine

Disclosures

– UK Commission for Human Medicines– EMA Neurology SAG

– UK Animals in Science Committee

– Independent Statistical Standing Committee, CHDI Foundation

– Project co-ordinator, EQIPD IMI


I am not in the office at the

moment. Send any work to be

translated.


Winner of the 2012 Ignoble Prize for Neuroscience


Control half dose full dose

Infa

rct V

olum

e

0

50

100

150

200

250

300

10-120 M 10-60 M

Presenter

Presentation Notes

Get screenshot of paper


1026 interventions in experimental stroke

O’Collins et al, 2006

In vitro and in vivo - 1026

Tested in vivo - 603

Effective in vivo - 374

Tested in clinical trial - 97

Effective in clinical trial -1


• NXY-059: Infarct Volume– 11 publications, 29 experiments, 408 animals– Improved outcome by 44% (35-53%)

Macleod et al, 2008

Effi

cacy

Randomisation Blinded assessment of

outcome

Blinded conduct of experiment

Risk of bias in animal studies


You can usually find what you’re looking for …

Group Day 1

Day 2

Day 3

Day 4

Day 5

“Maze bright”

1.33 1.60 2.60 2.83 3.26

“Maze dull”

0.72 1.10 2.23 1.83 1.83

Δ +0.60 +0.50 +0.37 +1.00 +1.43

Rosenthal and Fode (1963), Behav Sci 8, 183-9

• 12 graduate psychology students• 5 day experiment: rats in T maze with dark arm alternating at random, and the

dark arm always reinforced• 2 groups – “Maze Bright” and “Maze dull”


Evidence from various neuroscience domains …

Blinded assessment of behavioural outcomeNo Yes

Impr

ovem

ent i

n be

havi

oura

l out

com

e (S

tand

ardi

sed

Effe

ct S

ize)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Multiple Sclerosis Parkinson´s disease

Alzheimer´s diseaseStroke


1173 publications using nonhuman animals, published in 2009or 2010, from 5 leading UKuniversities

“an outstanding contribution tothe internationally excellentposition of the UK in biomedicalscience andclinical/translational research.”

“impressed by the strength within the basic neurosciences that were returned …particular in the areas of behavioural, cellular and molecular neuroscience”

The scale of the problem: RAE 1173

Rand Blind I/E SSC

Presenter

Presentation Notes

We were told it was unselected institutions


BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

BEFORE

AFTER

Impact of NPG checklist


In vitro experiments

0% 20% 40% 60% 80% 100%

in vivo

in vitro

Randomisation0% 20% 40% 60% 80% 100%

in vivo

in vitro

Blinding

0% 20% 40% 60% 80% 100%

in vivo

in vitro

Power calculation

Full Partial Null

0% 20% 40% 60% 80% 100%

in vivo

in vitro

Reporting exclusions

Presenter

Presentation Notes

747 in vivo, 640 in vitro


IICARUS (PI Sena)

Protocol: Open Science Framework (February 2017)Data Analysis Plan: Open Science Framework (September 2017) Funding: MRC, NC3Rs, BBSRC & Wellcome TrustEthics: BMJ Ethics Committee

Normal Handling

1013820

Web based Crowd sourced Assessors trained Dual ascertainment Reconciliation by third reviewer

845

844

Ran

dom

ised

340

322

Acce

pted

301

13

Che

cklis

t com

plet

ed

0

0

ARR

IVE

com

plia

nt


Reproducibility and replication

“Reproducibility” related to the re-analysis of existing data following the same analytical procedures.

“Replication” was held to require the collection of new data, following the same methods.


Replication studies

1. Retrospective – Pharmaceutical companies sharing their historical experience when they have attempted replication

– Bayer 33% of 67– Amgen 11% of 53

Selection bias (2 companies out of ?)? Recall Bias? What did they replicate? How?


Replication studies

2. Prospective - Academic led, great attention given to faithfulness to original study design, adequate statistical power, preregistration

– Psychology 36% of 97 ESR=49%– Cancer biology 40% of 10– Economics 61% of 18 ESR=66%– Social sciences 62% of 21 ESR=54%

? Selection bias (how did they choose what to try to replicate?)


Claim

• Lack of reproducibility of experimental findings has been observed across such a wide variety of settings that it can be considered a general phenomenon

• Therefore, unless a field can demonstrate that it doesn’t have a problem, it is reasonable to expect that it does


Both studies may be correct[Reaction norms, Voelkl 2016]

Res

pons

e

“Nuisance” variable


Lifespan in worms

Lucanic et al Nature Comms 2017

Source of variation

Developmental Rate

Fertility

Genetic 83.1% 63.3%Between labs 8.3% 7.9%Within labs 3.8% 5.6%Individual 4.8% 23.3%

Presenter

Presentation Notes

At the outset of this project we noted extensive differences in details of media preparation, bacterial growth, animal handling and experimental protocols when we compared working practices among CITP labs. Although a fully ‘robust’ intervention might withstand myriad protocol differences, we recognized that interpretation of mixed or negative results would be clarified only if we conducted experiments as similarly as possible. We found that multiple C. briggsae strains exhibited a striking bimodal survival outcome in all three labs. For example, a total of 25 survival assays were undertaken on HK104 across the three labs. We observed that 13 cohorts were relatively short-lived and 12 cohorts were long-lived. We observed this bimodality in almost all C. briggsae strains to some degree. The bimodal pattern appears to be a systematic source of within-lab variance intrinsic to the system. Strong evidence supports that even single-celled organisms can adopt distinct physiological states47. Our findings raise the question as to whether C. briggsae might have the capacity to transit through distinct physiological states according to unidentified cues. The Caenorhabditis capacity to adopt distinct physiological states could also account for some of the reported heterogeneity in behaviours and phenotypes.



Trans-lational research


Cis-lational research


If Σ(knowledge) < threshold cis – lation

If Σ(knowledge) > threshold trans - lation


Researchers are different …

num

ber

quality

FalsificationFabricationPlagiarism

HARKing

Preregistration

Risks of bias

Open Science


How we do research integrity


Research Improvement Strategy


Edinburgh Research Improvement Project

IRB/AWERB Approval Conduct Publication

ImprovementOpportunity

InstitutionalRepositoryEvaluation

33 items across 5 domains:

DOMAIN 1: exploratory vs. confirmatory researchDOMAIN 2: pre-planning and SOPsDOMAIN 3: statisticsDOMAIN 4: randomization and blindingDOMAIN 5: documentation


Recommendations

• Every institution which conducts research should have a formal Research Improvement Strategy

• Funders should consider the high returns on Replication studies (30-50% chance of revising existing knowledge)

• Pivotal research findings should be challenged in prospective multicenter replication studies prior to exploitation


Biomedical research investment

• $300bn globally, €50bn in Europe• Glasziou and Chalmers claim 85% wasted• Even if waste is only 50%, improvements which reduced

that by 1% would free $3bn globally, €500m in Europe, every year.

• Investing ~1% of research expenditure in improvement activity would go a long way


If you are planning a systematic review or meta-analysis of animal data, CAMARADES and SyRF are here to help.

Re-imagining failure when conceptualising research ... · vid Moher,, Davina Ghersi, Douglas G Altman, Elaine Beller, Elina Hemminki, ElizabethWager, Fujian Song, Harlan M Krumholz,

Documents