The Reality of Reproducibility in Computational Science · • Marco Roos • Jose Enrique Ruiz del Mazo • Oscar Corcho • Anton Güntsch • Cherian Mathew • Ian Cottam •

The Reality of Reproducibility in Computational Science reproduce? repeat? rerun? does it matter?

Prof Carole Goble FREng FBCS The University of Manchester, UK [email protected]

Based on: e-Science 2012 Chicago, October 2012 https://dl.dropbox.com/u/617206/eScience-2012-GOBLE-release-nonotes.ppt

JCDL 2012 Washington DC, June 2012 https://dl.dropbox.com/u/617206/JCDL%20Goble%20Final%20Clean-nobigbird.ppt

Scholarly Communication Workshop, 14-15 January 2013, Pittsburgh, USA

Computational Methods Scientific workflows. In the wild. Distributed web/grid/cloud services Cyber-Infrastructure

Social Methods: Sharing and Exchange e-Laboratories for scientific artefacts. Libraries, Repositories and Catalogues for data, models, web services, workflows, scripts, SOPs…

Knowledge Management Semantic technology, semantic applications, Linked Open Data, research objects, executable papers, publishing

Software Engineering Software Sustainability Institute Open Middleware Infrastructure Institute, S/W and Data Policy Institutional Repository

Systems Biology

Chemistry

Astro-Physics

Astronomy

Biology

Social Science

Library Digital

Preservation Biodiversity

Public Health

Products Methods Applications

Research Objects

Nanopublications

Systems Biology data, models and SOPs

Service and Workflows

Data, Service and Workflows

Reproducibility a principle of the scientific method

Evidence to test and justify claims

Comparison of results and methods

Peer review

http://xkcd.com/242/

“An experiment is reproducible until another laboratory tries to repeat it.”

Alexander Kohn The Reproducibility Initiative

Reproducibility as a Service PLoS, FigShare

http://reproducibilityinitiative.org

Data Journals / Repositories

In silico (Computational) Science

Datasets Data collections Algorithms Configurations Tools and Apps Codes Workflows Scripts Code Libraries Services, Infrastructure, Compilers Hardware

Simulations, data exploration, data processing, analytics, database based, text mining, auto recommendation, visual analytics…(Digital Science = Science)

Science 13 April 2012: 336(6078) 159-160 DOI: 10.1126/science.1218263

DASTY Ensembl Browser JWS Online Simulator

Specialist Codes Libraries, Platforms, Tools

Service based Science

(Cloud) Hosted Services

Cytoscape

Commodity Platforms

Data Collections Catalogues Software

Repositories

My Data My Process My Codes My Libraries My Special Tweaks

Compound Assemblies: Workflows See Tom Moritz talk Execution

Multi-step coordinated execution of (distributed) computational components Repeatable and comparative Explicated computation

Virtual Witnessing / Minute-Taking Transparent, precise, citable documentation Accurate logs Reusable protocols, know-how, best practice

nameComplete Ameira divagans Boccardia redeki Bougainvillia rugosa Branchiura sowerbyi Cercopagis pengoi Chelicorophium curvispinum Chionoecetes opilio

courtesy Matthias Obst, University of Gothenburg, Sweden!http://www.biovel.eu!

Study on the ecological niche of the south east Asian horseshoe crab

•  Generate input files: Import south east Asian data from public archives + Clean data + Merge with own data

•  Run large number of niche model analyses •  Visualise ecological niche maps to interpret and

compare

Local & 3rd party independent resources Shielded heterogeneous infrastructures

BioSTIF

Components, Services

Platform Libraries and

Plugins

Workflow Description

Datasets, Parameters

Configurations

Assemblies Dependency

Reproducibility Issues Read It: Description •  Obfuscated: too vague / detailed •  Black Box data/processes •  Tweaking •  Scattering •  Logging

Run it: Environment •  Dependencies/Stewardship •  Stability/Reliability •  Availability: one-off processes •  Black box platforms •  Scattering •  Tweaking •  State: Snapshot or Live

Zhao, Gomez-Perez, Belhajjame, Klyne, Garcia-Cuesta, Garrido, Hettne, Roos, De Roure and Goble. Why workflows break - Understanding and combating decay in Taverna workflows, 8th Intl Conf e-Science 2012

Do It: Governance •  Capability •  Cost / Burden •  Credit / Reward

Laboratory

Instruments

Methods

Materials Publica(on  

Models,  Techniques,  Algorithms  

Data  

Laboratory

Instruments

Methods

Materials

Provenance  A;ribu(on  Credit  

Context  Inves(ga(on  

Study  Experiment  

Replicate / Repeat Exactly replicate the original experiment and experimental conditions. Eliminate change. Observe.

Reproduce Run experiment with differences in experimental conditions.. Compare to test for same result. Observe.

Capture Curate Discover Use Reuse Preserve

Reproduce Between Labs

Repeat Within Lab

Reproduce

Replicate Repeat

Verify

Recreate results without existing code or data, independently.

Re-run to determine the sensitivity of results when underlying measurements are retaken

Regenerate results from existing code, data.

(Re)examine accuracy, wrt underlying model (Verify), or data (model error, measurement error) (Validate)

Adapted from V. Stodden, “Trust Your Science? Open Your Data and Code!” Amstat News, 1 July 2011. http://magazine.amstat.org/blog/2011/07/01/trust-your-science/

Re*<verb> Bingo

Fix and Compare

Vary and compare

Review the Record

[adapted from W

atson and Missier]

Decay

Reproduce Repeat

Replicate

Version Control (data, services, workflows)

No Version Control

External Dependencies Mixed Environments (open service set)

Complete control over services Single Environments (closed service set) Enclaves

Workflows in the Wild

Detect and Repair

Dependencies Snapshots

Community Workflows

Virtual Machines, Deployed Codes Prevent

Reproducible Research Systems There are many emerging (time for “standards”?)

•  ID it to Cite It: ORCID (people), DOI (data, models, tools ...) •  Tracking: local helper systems to instrument and track

provenance •  Science as a Service: Virtual Machines, Cloud Appliances,

Hosted platforms deploys on your behalf, no installations, common platforms (e.g. Galaxy)

•  Libraries and Repositories: with rich documentation •  Publish: executable papers, companion web sites,

embedded notebooks/publishing, active publications •  Explication of experimental mechanics: pipelines, workflows,

script systems with documentation, common tools/languages (e.g. MatLab)

Research Objects

•  Technical –  Preservation & Restoration –  Provenance Tracking –  Executable Publication

•  Social –  Unit of Scholarship –  Preservation protocols –  Credit Tracking

•  Semantics –  Publishing, Exchange –  Aggregated Carriers of Research Context

Capture relationships between people, papers, data and analysis protocols

Distributed Third Party Tenancy

Alien Store

http://www.wf4ever-project.org

http://www.myexperiment.org/packs/231.html

1. Reproducibility is a means to an end, not an end in itself

•  all science becomes less reproducible / repeatable over time…

and some can never be

… stochastic experiments or large scale data collections.

•  when does it matter?

icanhascheezburger.com

Results may vary

Defend results are correct and method convincing and repeatable.

Review & Learn Verify the results empirically. Trust. Understand. Convince, comfort, credibility.

Reuse Use the explained and trusted results (data, method) for new / my science on demand. Compare. Extend.

Is it “true”? Can I repeat it?

Can I use it? Can I reproduce it?

Method Provenance (link data and code)

Data Method Documentation

Method Execution

Snapshot State Available

Replay

Recreate

Altered State Available

Rerun Repeat

Reproduce with new Data

Reproduce with new Method

Repair Documented Provenance Of State

Recover Repurpose

Reuse Review

Good enough To Verify

Drummond C Replicability is not Reproducibility: Nor is it Good Science, online Peng RD, Reproducible Research in Computational Science Science 2 Dec 2011: 1226-1227. De Roure http://www.scilogs.com/eresearch/replacing-the-paper-the-twelve-rs-of-the-e-research-record/

Method

Reproduce Method

Extend

2. Reproducibility is a Spectrum

Replicate

Reproducibility is a Spectrum Partial reproducibility – over proprietary steps

or difficult-to-reproduce subparts, or just through examining the log

“perfect is the enemy of the good” Voltaire

3. Reproducibility through Inspection Archived Record to Manage

d1

S0

d2

S1

w

S2

y

S4

df

d1'

S0

d2

S1

z w

S'2

y'

S4

df'

(i) Trace A (ii) Trace B

http://ww

w.w

f4ever-project.org/research-object-model

Log, Fix, Replay, Analyse -> Instrument Systems and Apps

[Woodman, et al, 2011]

W3C PROV

4. Reproducibility by Invocation Active Instrument to Maintain

•  Active Preservation: –  Preservation vs Just in Time Just Enough

restoration/reconstruction: The natural state is broken.

•  Stop Publishing, Start Releasing –  Software release practices for workflows and

scripts, services, data and articles [Schopf, JCDL 2012]

•  Librarianship, Stewardship and Best Practices of Everything –  “Better Science through Superior Software” – C

Titus Brown –  Zeeya Merali , Nature 467, 775-777 (2010) | doi:

10.1038/467775a

Data Stewardship

Software sustainability Software practices Software deposition Long term access to software Credit for software Software Journals Licensing Open Source Software

Best Practices for Scientific Computing http://arxiv.org/abs/1210.0530 Stodden, Reproducible Research Standard, Intl J Comm Law & Policy, 13 2009 Prlić A, Procter JB (2012) Ten Simple Rules for the Open Development of Scientific Software. PLoS Comput Biol 8(12): e1002802. doi:10.1371/journal.pcbi.1002802

Software “Better Science through Superior Software” – C Titus Brown

Open does not mean understandable.

José Enrique Ruiz (IAA-CSIC)

Galaxy Luminosity Profiling Why?

Make it Matter. Trade, Asset and Curation economics

What? Numerous standards: formats, terminologies and checklists

When? Incremental, Eager and Lazy, UpStream, Downstream

How? Ramps: Automation & Integrated Tools

Who? Copy editing Method, Curation Service, Authors? Reviewers? Editors? Readers? Curators?

35 different kinds of annotations 5 Main Workflows, 14 Nested Workflows, 25 Scripts, 11 Configuration files, 10 Software dependencies, 1 Web Service, Dataset: 90 galaxies observed in 3 bands

5. Governance, Economics and Burden

Publica(on  

Models,  Techniques,  Algorithms  

Data  

Provenance  A;ribu(on  Credit  

Context  Inves(ga(on  

Study  Assay  

Accessible Capable Reusable

INSTRUMENTs Samples,Specimens Strains

74% / 26%

31% / 8%

See Fran Berman Talk

Hugging Flirting

Voyerism Creeping

Comprehending

Trading Credit

Economics Tenopir C, Allard S, Douglass K, Aydinoglu AU, Wu L, et al. (2011) Data Sharing by Scientists: Practices and Perceptions. PLoS ONE 6(6): e21101. doi:10.1371/journal.pone.0021101

See Bill Arms talk http://biocurator.org/

Integrated Reproducible Research Systems that the 95% use

•  Active Reproducible Research Environment –  Instrumented infrastructure and services

for producing and working with reproducible research.

•  Active Reproducible Research Publication Environment –  Instrumented infrastructure and services

for distributing and reviewing; academic credit; legal licensing, watching and preserving etc.

•  Safe Havens, Rescue Teams, Scholarship Services

•  Top Down and Bottom Up

* Adapted from Mesirov, J. Accessible Reproducible Research Science 327(5964), 415-416 (2010)

Design

Execution

Result Analysis

Publishing

Logbook Scientist

Collection

A GAP

Ramps for Authoring & Reading

http://www.rightfield.org.uk

UTOPIA Documents www.getutopia.com

Semantics, Thursday, 10:30–12:00

Wolstencroft, Owen, Goble, Nguyen, Krebs and Müller. RightField: Semantic Enrichment of Systems Biology Data using Spreadsheets

Governance, Ecosystems and the Scholarly Process

•  Local or Central responsibility •  Responsibility/Role/Reward of:

–  Institution? Funders? –  Library? Publishers? –  Reviewers? Trainees? –  Authors? Readers? –  Communities? Curators? –  Information Brokers? –  Third party vendors? –  Research Management service

providers? •  Cost/Capacity/Reward for review •  Sustainability, Silos, Packaging •  The 95% http://reproducibilityinitiative.org

Its harder than you might think. And less common than it could be.

Its about capturing, preserving, reusing and curating.

Bottom Up Perspective

“An experiment is reproducible until another laboratory tries to repeat it.”

Alexander Kohn

Summary •  Couple together Library,

Infrastructure, Publishing, Culture, Social, Policy

•  Reproducibility for the 95% •  Bottom up not just top down

•  “Weak” reproducibility is better than none at all and could be enough.

Acknowledgements and Inspirations •  David De Roure •  Tim Clark •  Sean Bechhofer •  Robert Stevens •  Christine Borgman •  Victoria Stodden •  Marco Roos •  Jose Enrique Ruiz del Mazo •  Oscar Corcho •  Anton Güntsch •  Cherian Mathew •  Ian Cottam •  Steve Pettifer

•  Wf4ever, SysMO, BioVel, UTOPIA and myGrid teams

•  Robin Williams •  Pinar Alper •  C. Titus Brown •  Greg Wilson •  Juliana Freire •  Jill Mesirov •  Simon Cockell •  Paolo Missier •  Paul Watson •  Gerhard Klimeck •  Matthias Obst •  Jun Zhao •  Pinar Alper •  Daniel Garijo •  Yolanda Gil

Further Information •  myGrid

–  http://www.mygrid.org.uk •  Taverna

–  http://www.taverna.org.uk •  myExperiment

–  http://www.myexperiment.org •  BioCatalogue

–  http://www.biocatalogue.org •  SysMO-SEEK

–  http://www.sysmo-db.org •  MethodBox

–  http://www.methodbox.org.uk •  Rightfield

–  http://www.rightfield.org.uk •  UTOPIA Documents

–  http://www.getutopia.com •  Wf4ever

–  http://www.wf4ever-project.org •  Software Sustainability Institute

–  http://www.software.ac.uk •  BioVeL

–  http://www.biovel.eu •  Force11

–  http://www.force11.org •  http://reproducibilityinitiative.org •  http://reproducibleresearch.net