Hello Sensor 1
Hello Sensor
1
Agenda1. Weather Underground Introduction 2. Making Your Own PWS 3. Data Ingestion & QC 4. API
2
Weather Underground Intro
3
What is Weather Underground?● Web● Flagship app● Storm● WunderStation● PWS Network● API
4
Web
● Powered by 200k+weather stations
● Visually engaging● Provides low-level
weather data
5
Flagship App
● The most hyperlocal forecasts● Data presented in a stunningly
simple interface
6
Storm
● The best app for the worstweather
● Highest resolution radar● Severe weather alerts
7
WunderStation
● Personalized weather dashboard
● Features your own PWS data
8
PWS Network● There are about 12k
government provided weather stations
● We fill in the gaps withover 200k PersonalWeather Stations
9
Making Your Own PWS
10
What is a Weather StationTraditional stationsQualitative reporting (crowd reports)Image recognitionPhone SensorsCar sensorsMaker Station
11
Weather hungry data monstersTo serve globally we need more data-Engage with local met offices (if they exist)-Engage with education/maker community
More data, better data = better forecasts.
12
Roll your ownOpen source weather stations make IoT and weather more available/flexible for local needs
Can be part of an education program
13
What does it take1.Sensor (Temp, precip, humidity, uv, etc)2.Controller (arduino, particle, etc) 3.Memory and/or Transmitter (flash,wifi, cellular)4.Power (solar, battery, mains)
14
Station challengesHardware:1:power (limits everything)2:transmit (expensive power budget item)3:durability (usually moving parts)
4:sensors (minor technical issues)5:controller (very low requirements)
15
Station challengesBiggest contributor to data variation: Enclosure design
The Effectiveness of the ASOS, MMTS, Gill, and CRS Air Temperature Radiation Shields: K. G. Hubbard, X. Lin, and E. A. Walter-Shea 16
Tiny wifiTiny wifi connected station
limited battery life
Used to monitor terrarium
17
Ol faithfulGood reliability, online over a yearSolar and battery powered
Enclosure made from ~$6 garden supplies
Particle Photon (WiFi)
Spark Fun Weather Shield -HTU21D humidity sensor-MPL3115A2 pressure sensor
18
Cell-o thereParticle Electron: cell radio + microcontroller BMP280: temp, humidity, pressure sensorEnclosure made from a painted soda cup
Data is good if kept in shade however: no venting = heat buildupok proof of concept, needs refinement
19
Data Ingestion & QC
20
Ingestion
Rapidfire● Ingests and stores data reported at rates as fast as one observation
every 2 seconds● Stores data in current condition file, records history data at as high
resolution as once every 5 seconds
21
Quality Control (QC)Before QC
22
Quality Control (QC)After QC
23
Quality Control (QC)
24
The QC Checks
● Range Check● Stuck Sensor Check● Neighbor Check
25
Range CheckHave these readings ever happened on Earth?
Temperature < -130º F or > 135º F.Dew Point < -90º F or > 90º F.
Wind Speed < 0 mph or > 279 mph. Wind Direction < 0º or > 360º.
Pressure < 846 inHg or > 1100 inHg.
26
Stuck Sensor CheckHas the temperature changed in the past 6 hours?
● by at least 0.1°F● lack of change is often an indication of
other stuck sensors as well
27
Neighbor CheckIs the temperature of this station similar to the majority of stations nearby?
● collect sensors in 15 km of current sensor● find clusters divided by 3° F● determine majority cluster(s)● throw out statistical outliers
Most essential customer-facing check
28
Neighbor Check
29
The Next Step - QC on Ingest● Current QC
○ cycle is 15 minutes, allowing bad observations to linger on the site and apps during that time
○ written in difficult to maintain and extend multi-threaded C++ code
● IBM Streams + QC○ clean obs all the time○ written in single threaded Python with better performance, stability,
extensibility, third-party libraries like Spark, and support for modern technologies like JSON and REST
30
API
31
200,000+ Personal Weather Stations
2.2 Billion forecast locations | 180 M consumers / month 32
33
Uptime: 99.95 % Latency ~25 ms
Autoscale to 20B requests per day
Scalability Average 10s of Billions requests per day
Global Coverage (US East, US West, EU, Asia)
Partial DeploymentsVersioned artifacts and rollbacks
Faster code to prod: Less dependency b/w teams
Your favorite tech / language here
34
Architecture: Storage Polyglot
Real time data and caching
Historical weather data Data Migration
Gateway Data Analytics
ArchivesImagesVideos
AnalyticsInformatica
Drupal
35
Thank you!
36
Questions?
37