NETWORKED MACHINE LEARNING JOAQUIN VANSCHOREN (TU/E),2014 #OpenML
Jun 29, 2015
N E T W O R K E D M A C H I N E L E A R N I N G
J O A Q U I N VA N S C H O R E N ( T U / E ) , 2 0 1 4
#OpenML
Research different.
1 6 1 0
G A L I L E O G A L I L E I D I S C O V E R S S A T U R N ’ S R I N G S
‘ S M A I S M R M I L M E P O E TA L E U M I B U N E N U G T TA U I R A S ’
Research different. Royal society: Take nobody’s word for itScientific Journal: Reputation-based culture
3 0 0 Y E A R S L AT E RJ O U R N A L S S H O W L I M I T S
• Complex code not included
• Large data sets not included
• Experiment details scant
• Results hard to reproduce
• Papers not updatable
• Slow, incomplete tracking of
paper impact
• Publication bias
• No online public discussion
• Open access?
J O U R N A L S : L O N G - T E R M M E M O R Y I N T E R N E T: S H O R T- T E R M W O R K I N G M E M O R Y
N E T W O R K E D S C I E N C E
O N L I N E D A TA B A S E S
O P E N S O U R C E C O D E
W E B S E R V I C E S , A P I S
C O L L A B O R A T I V E T O O L S
!
O P E N , S C A L A B L E C O L L A B O R A T I O N
R E A L - T I M E D I S C U S S I O N
C O M B I N E , R E U S E S C I E N T I F I C R E S U LT S
C I T I Z E N S C I E N C E
Research different. Polymaths: Solve math problems through massive collaboration (not competition)
Broadcast question, combine many minds to solve it
Solved hard problems in weeks
Many (joint) publications
Research different. SDSS: Robotic telescope, data publicly online (SkyServer)
+1 million distinct users vs. 10.000 astronomers
Broadcast data, allow many minds to ask the right questions
Thousands of papers
Research different. Galaxy Zoo: citizen scientists classify a million galaxiesOffer right tools so that anybody can be a scientistMany novel discoveries by scientists and citizens
Research different. Sharing data sparks discovery
Designed serendipity: - What’s hard for one scientist is
easy for another - Surprising ideas, observations
can spark new discoveries
Share, organise data for easy, large-scale collaboration
Data exploding in all sciences: collaborative data analysis needed
Building reputationAuthorship: easy to contribute + contributions stored, visible online
Collaboration: build trust, work with new people
Citation: more people see, build upon, and cite your work. Tell people how to cite data and code.
Altmetrics: track reuse/interest online (ArXiv)
N E T W O R K E D M A C H I N E L E A R N I N G
Machine learningComplex code, large-scale data, experiments (impossible to print)
Experiments not shared online: impossible to build on prior work: inhibits deeper analysis (e.g. meta-learning)
Low reproducibility, generalisability (studies contradict)
What if we could all connect with each other, and with other scientists, to explore and apply machine learning?
Few collaborative tools to speed up research
OpenMLPlace to share data, code, experiments in full detail
All results organised, linked together for further (meta)analysis, reuse, discussion, study, education
Links to (open-source) code, open data anywhere online.
Anyone can post data to analyse, anyone can share code and results (models, predictions, evaluations)
Integrated in ML platforms (R, Weka, Rapidminer,…) to automatically load data, upload results
Scientists can work in teams, but results only publicly visible if data, code shared
OpenML: benefits for scientistsMore time: automates routinizable work: - find data and/or code - setup and run large-scale experiments - results compared to state-of-the-art - log experiment details for future reference
More control: - state how others should cite your work - track reuse - share results more easily
More knowledge: - more time for actual research - build directly on prior work - easier, large-scale collaboration + interaction
Plugins: WEKA
Plugins: MOA
Plugins: RapidMiner
1 . O P E R AT O R T O D O W N L O A D TA S K ( TA S K T Y P E S P E C I F I C )
2 . S U B W O R K F L O W T H AT S O LV E S T H E TA S K , G E N E R AT E S R E S U LT S
3 . O P E R AT O R F O R U P L O A D I N G R E S U LT S
OpenML: under developmentOpenML studies - collection of datasets, flows, runs, results in a study - online counterpart of paper (with url) - construct by simply tagging resources - easily include (build on) data of others
Reputation building - Profile page: statistics of activity and impact on OpenML - Collaborative leaderboards: best contributors to solving a task
Teams - Add scientists in teams (circles) - Share resources, results within team only - Make public at any time (e.g. after publication)
Meta-learning support - Data/Flow qualities: easy adding, better overviews - Algorithm selection techniques running on website (vs humans?)
J O I N T H E C L U B