open access, p-hacking, registered reports, registered replication reports

Macquarie University 4 Dec 2015

A new era of large-scale replication and collaboration?

[email protected]

@ceptional

http://www.slideshare.net/holcombea/

• Open access

• P-hacking

• Preregistration, Registered Reports

• Open data, open methods

• Registered Replication Reports

Macquarie University 4 Dec 2015 2


Open access - Research funders will mandate it. Will we scholars shape the system of the future, or passively let publishers do it in their own interest?

• Green or Gold

• Who pays?

• Institution, granting agency, library (from subscription savings)

• Everything technically adequate should be published

• “contribution to the field” is subjective

• Competition for Nature on down wasteful, distorting

• Frontiers? PeerJ? Ubiquity Press?


Our average base Article Processing Charge (APC) is only £300 (€375, $500) and this is the only thing we charge.

5Slide courtesy Chris Chambers (Cardiff)

Science publishing has an incentive problem


"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.

#AcademicNoir


P-hackingSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-

positive psychology undisclosed flexibility in data collection and

analysis allows presenting anything as significant. Psychological

Science


False positive zones - researcher degrees of freedom high (p-hacking)

- victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant.

Where science is not self-correcting

The newer the domain, the more cutting-edge the technology, the less-defined the expected analyses, and the greater the opportunities for p-hacking


courtesy Chris Chambers (Cardiff)


Well, Mr. Hacker, your luck is about to run out. Down at the journals, they’re running a new game. They call it ‘preregistration’.

CC-BY Alex Holcombehttps://twitter.com/ceptional/status/663850868396044288






https://osf.io/tvyxz/wiki/home/





100% P-hacking FreeHere, check our numbers.

Here’s how you can replicate our result.

The badges that are fixing science


"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.


Failed to replicate my result?

Have you considered

the unknown moderators?


Any one of us alone, he’ll say we messed up. But if we all run the replication together, we

can show the big guy we’re right.


Meta-analysis forest plot


Meta-analysis forest plot

Failed to replicate my result?

Have you considered the unknown moderators?

You messed up!





Registered studiesLarge-scale replicationOpen dataOpen protocol and code

A new era of large-scale replication

and collaboration?
