Code publication and peer review Peer review...Reviewers are provided private access to the capsule and free-computing time. Readers access code, data and environment in one place,

Code publication and peer reviewa discussion of best practices, technological innovations and recognition

Erika Pastrana, Editorial Director,

Nature Journals

An

tarctica meltd

ow

n co

uld

do

ub

le sea level rise

Outline

What problem are we trying to solve?

Best practices and guiding principles

Technological innovations---Trialing container platforms for peer review and publication of code. Results from Code Ocean-Nature journal trial.

Incentives and Recognition

Discussion

2

Nature Research PowerPoint presentation title / 00.00.2017

2

1.0

What problem are we trying to solve and what are our guiding principles?

Today’s scientific discovery process depends on a dynamic set of digital resources: data, code and the right environment to run the analysis.

The number of research papers that make use or develop new code is increasing rapidly.

When code is central to the main claims in the paper, it is imperative that the code be properly documented, evaluated so that it is functional, and permanently accessible to ensure reproducibility of the results.

As publishers, we want to adapt to the changing outputs of science, improve our papers to increase reproducibility and open science, and lead in innovation.

What problem are we trying to solve?

4

Nature Research PowerPoint presentation title / 00.00.2017

Best practice and guiding principles

• Proper documentation : Code needs to be sufficiently documented to enable others to check and re-use it. This includes information on dependencies, operating systems, technical requirements as well as licenses and terms of use.

• Peer review: peer reviewing the code ensures that it is evaluated by an expert and it is functional and re-usable.

• Sharing and recognition : as the code is a central element of the paper and the research output, it should be shared and cited via a permanent identifier (DOI) in a form that facilitates wider use by the academic community and recognition of its own value.

5

Nature Research PowerPoint presentation title / 00.00.2017

5

2.0

Technological innovations

A case study: Nature Journals trial the use of container platforms for peer review and publication of code

• Currently, the Nature journals mandate that the availability and conditions of access of custom code that is central to the paper, be explicitely stated.

• For over 10 years, several Nature journals (eg Nature Biotechnology, Nature Methods)have been asking authors to submit the code for peer review and asking reviewers to check it by installing and running the code. These journals also mandate that the code be shared in the published the paper.

Several Nature journals have a long history in making code accessible and robust upon publication

Reproduced from Pastrana, E., Kousta, S. & Swaminathan, S. Three Approaches to Support Reproducible Research. Science Editor

7

Nature Research PowerPoint presentation title / 00.00.2017

But….…publishing reproducible code is cumbersome and time consuming

Reviewers have to…

• Find the code• Acquire the right hardware• Set up the environment• Import the right data files• Install dependencies, packages, versions, operating systems…• Errors… fix… errors…• Run (on own compute time)• Verify code runs and produces results

+

• Contrast with claims in the paper

8

Nature Research PowerPoint presentation title / 00.00.2017

http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication

A trial to test the use of container platforms for peer review and publication of code

We want to adopt improved technologies for peer review and publishing of code:

Software container tools, such as Code Ocean (https://codeocean.com/) are cloud-based platforms that host the code, data and environments necessary to re-run the code and reproduce the results shown in the paper.

Users can run the code with the set parameters to enable reproducing the results or execute the code with new input values.

We ran a trial (August 2018 to Sept 2019) on Nature Methods, Nature Biotechnology and Nature Machine Intelligence in partnership with Code Ocean, to test the usefulness of code containers (aka ‘code capsules’) for peer review and publication of code.

Trial details:

Authors are given the option to set up a Code Ocean capsule when paper goes out to review to facilitate compliance with code peer review and publication policies.

Reviewers are provided private access to the capsule and free-computing time.

Readers access code, data and environment in one place, via a link to the capsule. Capsule is given a DOI to enable proper recognition, citation and re-use of code.

A trial to test the use of container platforms for peer review and publication of code

11

Nature Research PowerPoint presentation title / 00.00.2017

Reproduced from Pastrana, E., Kousta, S. & Swaminathan, S. Three Approaches to Support Reproducible Research. Science Editor

12

Nature Research PowerPoint presentation title / 00.00.2017

12

3.0

What have we learnt from the trial?

13

Nature Research PowerPoint presentation title / 00.00.2017

What we set out to monitor:KPI WhoMonitoring author opt-in to the trial and asking for feedback from authors if they decline. Editors

Effects on peer review: TAT from submission to first decision and TAT for CO capsule set up services

Editors/Code Ocean

Effects on peer review: qualitative assessments on the ‘quality’ of peer review of code Editors

Survey authors and referees involved in the trial to monitor author and referee satisfaction with the user interface, level of Code Ocean support and overall feedback on the use of the platform for peer review.

Editors

Engagement with Code Ocean capsule: Code OceanNumber of duplications of the compute capsule by reviewers Code Ocean

Total runs of compute capsules (pre publication) Code Ocean

Total runs of compute capsules (post publication) Code Ocean

Number of compute capsule views (post publication) Code Ocean

Total duplications/downloads of compute capsules (post publication) Code Ocean

Total numbers of citations to the code’s DOI (post publication) Code Ocean

Results: We now ensure our papers provide open, verified, properly

documented and cited code

Results:We see high author uptake despite the additional work required

Papers that participated in the trial >120 papers in the trial from the three journals

Papers published:>20 papers published that have used the functionality for peer review and publication of code

Author uptake:

54% of all authors from the three journals opted in to the trial, with 59% opting in at Nature Biotechnology, 44% at Nature Methods and 60% at Nature Machine Intelligence

16

Nature Research PowerPoint presentation title / 00.00.2017

Reviewer engagement :

On average private links were viewed 34 times. 45% of the capsules received a reviewer sign up/duplication On average, each reviewer that signed in run the code 1.3 times.

Reader engagement:

Published Code Ocean capsules associated with Nature papers were viewed on average 122 times and were run 14 times

(*) data from code ocean site.

Results: We see high reviewer and reader engagement with the platform

Reasons for opt out:

• 20% of authors who opted out did so because they felt their current platform (typically GitHub) already complied with our policies for code peer review

• 23% of authors thought Code Ocean would not be a suitable platform for their paper*

• 10% of authors feared that setting up a capsule would delay publication*

• 40% of authors who opted out gave no reasons

(*) potential to be reduced with better communication

Results: The main reasons for opt out are that authors are using GitHub

or think the platform won’t be suitable for them

Reviewer & Author comments

“I found the Code Ocean platform an excellent tool for reviewing the code thanks to its easy environment setup and clear instructions for running and data analysis. In order to review the code, I also carefully read the manuscript, so I offer my remarks for both”

“In addition to reviewing the manuscript, I tested its code and sample data through the Code Ocean container platform. I found the files and documentation provided with the package to be thorough, and was able to run the program without issue. Further, it was well-documented and easy to use (especially in the Code Ocean container system) and that will hold serious appeal to readers who might consider implementing this tool.”

“I do not have any experience with Code Ocean as a reviewer, but I would like to state that my experience with it as an author, it was straightforward for use by Authors and they were responsive and helpful when we reached out to them. So I have positive experience with them in that respect.”

“This is an interesting new tool, with useful features and reasonable amount of free computing time. It provides a great platform for code examination and dissemination, and should greatly contribute to reproducibility and open science.”

Results: We receive strong support for the initiative

19

Nature Research PowerPoint presentation title / 00.00.2017

Next steps and more information

We will continue with this initiative as a standard editorial practice going forward at

the existing Nature journals and are looking to expand this practice to more journals.

Nature Biotechnology: ‘Changing coding culture’

Nature Machine Intelligence: ‘Sharing high expectations’

Nature Methods: ‘Easing the burden of code review’

Science Editor: Three approaches to support reproducible research

Of Schemes and Memes: What we have learnt testing container platforms for peer

review and publication of code

Code Ocean blog post: Peer review on Code Ocean

20

Nature Research PowerPoint presentation title / 00.00.2017

20

4.0

Incentives and Recognition

21

Nature Research PowerPoint presentation title / 00.00.2017

Recognition

https://www.niso.org/press-releases/2019/01/reproducibility-taxonomy-definitions

• We would like to encourage best practice among the community by awarding papers that provide verified/peer reviewed code a ‘badge’ that follows community standards

• Badges will promote sharing code and will signal when code has been verified/peer reviewed

• We are part of a NISO working group developing Badge-recommendations for the computational and Computing sciences

22

Nature Research PowerPoint presentation title / 00.00.2017

22

The story behind the image

Antarctica meltdown could double sea level rise

Researchers at Pennsylvania State University have been considering how quickly a glacial ice melt in Antarctica would raise sea levels. By updating models with new discoveries and comparing them with past sea-level rise events they predict that a melting Antarctica could raise oceans by more than 3 feet by the end of the century if greenhouse gas emissions continued unabated, roughly doubling previous total sea-level rise estimates. Rising seas could put many of the world’s coastlines underwater or at risk of flooding and storm surges.

Thank you

Erika Pastrana, PhD

Editorial Director, Nature Journals

[email protected]

@ErikaPastrana1