Openness in research Experiences at Finnish Meteorological Institute 28.10.2016 Finnish Meteorological Institute Mikko Strahlendorff Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihi
Openness in researchExperiences at Finnish Meteorological Institute
28.10.2016
Finnish Meteorological InstituteMikko Strahlendorff
Roope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
01.05.2023 2
Background for my experienceMostly employed by Finnish Meteorological Institute Weatherforecasting 1995-1997 Public service development 1998--99 Commercial service development and head of it
2000-2003 Head of Informations system services 2004-
2007 Seconded Expert to European Commission
Earth Observation program 2008-2010 Ministry of Transport and Communications
Ministerial adviser 2011-2012 EU research policy and international
organisation 2012- Helping to build European Research
Infrastructure Consortias ICOS, EuroARGO, SIOS, COPAL
Executive commiittee member in Group on Earth Observation
Openness experience at FMI – Mikko Strahlendorff
Grown with the internet
01.05.2023 3
FOSS revolution and the internet• Researcher Linus Torvalds at Helsinki
university releases Linux as Free and Open-Source Software
• 25 years anniversary now and• 96% of all web-servers run Linux• 69% of all mobile devices run Android
(developed from Linux)
• Sharing ones work with a huge network of like-minded ensures high quality results faster than anything
• Motivation comes from the common utility of the result
Openness experience at FMI – Mikko Strahlendorff
• Openness requires extra efforts• Free and open doesn’t generate revenue
• How to motivate financing? How to share expenses fairly?
• Sharing efforts is making the list of contributors long and single persons are less visible
• How to divide the fame?Solutions:• Application of the common to many challenges in
many places• Analysis, understanding and education for sale• Fame to the whole system:
• Data production is publishing!• Each use of data should be rewarded!
• FOSS tools and internet technologies reduce costs
Challenges with Openness – rewarding effort
01.05.2023 4Openness experience at FMI – Mikko Strahlendorff
Data is a great common motivationo Analysis needs data and it gets better with
more datao Selling scientific data is rarely lucrative
and data lying in databases unused is only expenses - no gain
o Data is good to be collected in similar ways for similar applications – common interest naturally networks collectors
o Quality control and technical solutions are perfect tasks for sharing
o Data dissemination has costs that are usually not planned in research
o Tracking usage is good for development and for finance motivation
01.05.2023 5
01.05.2023 6
Finnish Meteorological Institute opened its data in 2013 after government decision for all the Finnish public sector.
Basically everything that FMI has property rights on was opened.
Data is provided freely in machine readable format over the internet.
Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Data set Description Time Interval
Estimated publish date
Weather Observations
Temperature, Wind, Humidity, Ground Temperature…
10 min Open,older data to be added
Sun Radiation UV, Short and Long Term Radiation…
1 min Open
Marine Observations
Waves, Sea Temperature, Sea Level…
1 h Open
Weather Radars Precipitation Rate, Precipitation Amount…
5 min Open, older data to be added
Lightning Thunder Strikes in Finland
5 min Open
Example Data Sets
01.05.2023 7Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Example of Data SetsData set Description Time Interval Estimated
publish dateReal Time Observations
Real Time Observations from specific location(s)
AWS 2010 – Soundings 1959 –Flashes 1998 – Sea Level 1971 – Waves 2005 –
Openolder data will be added
Climatological Observations
Dayly and monthly temperature mean and extreme values from weather stations
1959 - Open
Climatological Observations
Monthly temperature and precipitation rate mean values interpolated to grid
1961 - Open
Climatological Reference
Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth.
Reference seasons:1971-2000 1981-2010
Open
01.05.2023 8Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Example Data SetsData set Description Time Interval Estimated
publish dateWeather forecast model HIRLAM RCR
Point forecasts and grid data
Latest model run(4 times a day) 0…54 h
Open
Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data
Latest model run(4 times a day) 0...54 h
Open
Environmental Monitoring Facilities
Weather observation stations, radars…
2015
Aviation Observations
METAR 30 min open
Ground & mast observations
Special observations from ground and masts
2016 /Open
01.05.2023 9Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Example Data SetsData set Description Time Interval Estimated
publish dateAir Quality Observations
Air Quality Observations 1h open
Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen
Latest model run (once a day)0…96h
2015
HELMI Ice Model Ice forecast model Latest model run(4 times a day) 0...54 h
open
Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height
2 times a day open
Road Weather Observations (LIVI)
Road Weather Observations
10 min open
01.05.2023 10Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data Portal
Meta data
Data Models
Services
The very same data portal works as Open Data and INSPIRE portal.
01.05.2023 11Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Registrationo Registration is required to use View and Download
Services o Working email address is the only mandatory
informationo After registration the user gets an API key which have to
be added into all requestso GET parameter fmi-apikey=…&o Header fmi-apikey; …o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?
o One can create several API keys with one email
01.05.2023 12Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Usage LimitsWith one API key it’s allowed to
o do at most 20 000 requests per day to Download Serviceo do at most 10 000 requests per day to View Serviceo do at most 600 requests per 5 minutes to both services
o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily
o So, with one API key it’s allowed to load everything onceo View service can be used for testing but can not be used as a
back end for popular clients
01.05.2023 13Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
And a little over450 000 data downloads
per day (5.2 req/s)
At the moment about 11 700
registered users
3 year experience
01.05.2023 14Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Industry is happy to use standardized
services
Amateur and freelancer coders
would prefer simple JSON API
Some Experiences
01.05.2023 15Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
Please contribute!
https://github.com/fmidev/
MetoLibo Open source JavaScript library produced by Finnish
Meteorological Instituteo Helps users to load and use the data
o Supports multi point coverage data formato Python version is on the list
Easy requests Cache
Parse the data to as JSON
01.05.2023 16Meteorological and Aviation Weather Open Data implementation OGC standardsRoope Tervo, Mikko Visa, Tero Koivunen, Jukka A. Pakarinen, Tarja Riihisaari
01.05.2023 Open Meteorological Data with OGC and INSPIRE / Roope Tervo
17
SmartMet Server•High capacity & availability Data server for MetOcean data•Data is extracted and products generated always on-demand• INSPIRE Compliant•Operational use since 2008
• FMI client services (since 2008)• Finnish Meteorological Institute (FMI) Open Data Portal (since 2013)
•Several input sources• GRIB-, NetCDF-, etc. files via internal data format querydata
• PostGIS database (vectors)
• SQL database (point observations)
•Several output interfaces and formats• WMS, WFS 2.0
• JSON, XML, ASCII, HTML, SERIAL
• GRIB1, GRIB2, NetCDFC++
01.05.2023 Open Meteorological Data with OGC and INSPIRE / Roope Tervo
18
SmartMet Server goes Open Source•Going to be published as Open Source in Q4/2016•MIT license•https://github.com/fmidev
Fast scienceOpen Science visionScience WIKIPEDIA
o Data is collected in standard ways and published immediately
o Papers are written collaborativelyo Science is mostly about writing
scripts/models to analyze data for answers to questions
o Analysis scripts are published with the answers so readers can reproduce results and evaluate conclusions themselves
o Donors get to ask questions and use the knowledge
o Sharing the fame is visible thru linking all the scientists, from every data set and every model/script
01.05.2023 19Openness experience at FMI – Mikko Strahlendorff
• Governments: What data do we need for public interests?• How do private entities support data production?
• Science interest
• Commercial interest
• Research funding should be two-fold:• longer time plans and grants for data production and modeling
• short term grants for answers to societal questions for scripts and analysis
• How does a new Science approach conquer the world like Linux?
• Thanks and please comment!
What do we need to do for this vision?
01.05.2023 20Openness experience at FMI – Mikko Strahlendorff
www.fmi.fi
http://www.slideshare.net/MikkoStrahlendorffhttps://en.ilmatieteenlaitos.fi/open-data