Open science in cognitive modeling

PowerPoint Presentation

Open science incognitive modelingCaspar [email protected] @BrainStraining

International Conference on Cognitive ModelingUniversity of Groningen10 April 2015To the extent possible under law, Caspar Addyman has waived all copyright and related or neighboring rights to Open science in cognitive modeling. This work is published from: United Kingdom.

Open science is a bit more than open access.

Who has published in an open access journal?Who has shared data?Who has shared code?Who is publically funded?

http://creativecommons.org/publicdomain/zero/1.0/1OverviewHow I came to write a Modeling ManifestoWhat Ive done sinceWhat we should all do nextOverviewHow I came to write a Modeling ManifestoWhat Ive done sinceWhat we should all do nextWhat I am going to do nextTRACXFrench, Addyman & Mareschal, Psych. Rev. 2011

John Anderson Editor of Psychological Review made it a requirement that modellers share code but that didnt go far enough.4Truncated Recursive Autoassocative Chunk Extractor (TRACX)

Whats MATLAB?(French, Addyman & Mareschal, Psych Rev, 2011)AS modellers we might laugh at thisBut the question could easily be Whats JAVA? Whats R? EtcBayesian, symbolic, connectionist modelling conventions are unfamiliar to outsidersThere are more programming languages than there are natural languages..You feel like have to pick one.

5

http://leadserv.u-bourgogne.fr/~tracx/But no-one asks What is a web browser?So I built a web-based version of our model.Upside you can run it anywhere (even your phone)the downside was I had to write it twice6

http://github.com/YourBrain/TRACX-Web

Computational Modeling in Cognitive Science: A Manifesto for ChangeAddyman & French, TOPICS, 2012Let EVERYONE run your simulations (easily)Let motivated users adjust parameters/ run their own simulationsLet other modelers use your codeWrite good documentationClear mapping between model and empirical dataShow internal structureAllow import/export to Excel, JSON, etc 10Yes but..Implementations are not specifications Cooper and Guest, Cognitive Systems Research, 2013

1. Troubles with replicating modelling resultsGuest and Cooper (2012) ICCM 11

2. Bit rota consequence of continuing changes in technology code needs to be kept up to date.

3. Formal model specification ..more attention must be devoted to model specification as an independent exercise to implementation.

Playing ATARI with Deep Learning Google DeepMindNature 26 Feb 2015Released their codeIt is true though that we dont publish all our code right away. In fact, some of our code gets tied up in industrial projects which make it hard to release. Jurgen Schmidhuber, Reddit AMA, March 2015Psychology PhD students should know how to code. Jay McClelland

Opinions are divided about the need or value of making modelling and models accessible.But it is encouraging to see that googles 620million dollar algorithm factory share their secrets.12

http://convnetjs.com

Andrej Karpathy

13The associated source code of the software described by the paper. This should, as far as possible, follow accepted community standards and be licensed under a suitable license such as BSD, LGPL, or MIT. Dependency on commercial software such as Mathematica and MATLAB does not preclude a paper from consideration, although complete open source solutions are preferred.Documentation for running and installing the software. For end-user applications, instructions for installing and using the software are prerequisite; for software libraries, instructions for using the application program interface are prerequisite.A test dataset with associated control parameter settings. Where feasible, results from standard test sets should be included. Where possible, test data should not have any dependencies for example, a database dump.

Supporting information for submissions in which software is the central part of the paper:

Bianca Kramer & Jeroen Bosman(2015) dx.doi.org/10.6084/m9.figshare.1286826

101 Innovations in Scholarly CommunicationUtrecht University Library15

GAMIT-Net, CogSci 2014http://github.com/YourBrain/GAMITAddyman & Mareschal, 2014

In my most recent papers I not only release the code but I also include a function that with the right random seed replicates the figures in my paper. 16

Caspar Addyman. (2014). GAMIT Model MATLAB Code. Zenodo. dx.doi.org/10.5281/zenodo.9895Github has a one-click mechanism to make your repositories shareable.17GitHub

Caspar Addyman. (2015). Contec-Pulse-Oximeter-in-Matlab: Version 1. Zenodo. 10.5281/zenodo.16277

Caspar Addyman. (2015). Infant-Habituation-in-Matlab: Version 1. Zenodo. 10.5281/zenodo.16627So I did this for few snippets of code that Ive used. I cost me nothing to share them and benefitted people who encountered the same problem then googled a solution.18

If you use R to generate the figures in your papers then alongside the paper and the data share the script you wrote to make the figures. It makes your data a lot more tangible and comprehensible to other people at no extra cost ot you.19iPython / Jupyter

I believe notebook based solutions are the future.The ipython/Jupyter ecosystem is powerful enough for every modellers needs and comes with a nice way (nbviewer) to share beautiful, living documents that walk people through your model.20Beaker

beakernotebook.com

Beaker is another flavour of notebook that shows termendous promise. It is designed to be language agnostic which should help to make it future proof.And it does a great job of passing data from one language to another all in one place.Developed by smart Wall st quants for smart Wall St. quants. 21

Computational Modeling in Cognitive Science: A Manifesto for Change

Let EVERYONE run your simulations (easily)Let motivated users adjust parameters/ run their own simulationsLet other modelers use your codeWrite good documentationClear mapping between model and empirical dataShow internal structureAllow import/export to Excel, JSON, etc

??The biggest missing piece for psychologists, cognitive scienitists and cognitive modellers is a unified platform that lets us run experiments, models and statistics all on one platform. Web browsers are not great for running the type of tightly controlled experiments beloved of psychophysicistsPsychoPy is promising project in this regard.22

Computational Modeling in Cognitive Science: A Manifesto for Change

Let EVERYONE run your simulations (easily)Let motivated users adjust parameters/ run their own simulationsLet other modelers use your codeWrite good documentationClear mapping between model and empirical dataShow internal structureAllow import/export to Excel, JSON, etc

??The biggest missing piece for psychologists, cognitive scienitists and cognitive modellers is a unified platform that lets us run experiments, models and statistics all on one platform. Web browsers are not great for running the type of tightly controlled experiments beloved of psychophysicistsPsychoPy is promising project in this regard.23Try notebooksTry PythonTry sharing

24Questions for you1. What is your workflow? Is it connected?Is it shareable?If not, why not?

2. How can we narrow the gap between models and experiments?

[email protected]