The Critical Role of IoT Data Integration to develop Big Data Applications (for The Unmanned Systems Expo 2015, The Hague)

Rainer Sternfeld, CE September 2014

The Critical Role of IoT Data Integration to develop Big Data Applications

February 5, 2015

The Unmanned Systems Expo 2015, The Hague Rainer Sternfeld, CEO, Planet OS

@planet_osImage Credit: Northrop Grumman

2 February 2015

Sensor data will outgrow

social data in 2 yearsSensor data is

too big to moveData Management

will not scale as we know it

The bigger picture Swimming in sensors, drowning in data

3 February 2015

Hewlett Packard: “By 2020, 40% of all data

ever collected by human kind will be generated by sensors.”

4 February 2015

If you can’t use this data… what’s the point?

5 February 2015

Case #1: NOAA Real-time weather and climate data + forecasts

They are working hard on a challenge presented by:

• Tens of thousands of devices deployed in the ocean, on land, and space

• Tens of terabytes coming in every day

• 700 scattered “web services” (FTPs, flat files, Threads servers, no APIs)

• Breaking connections (all.the.time.)

• 25,000 employees

• Critical data for the government, the industry, and citizens

What if all this data would be easily discoverable and machine-readable?

6 February 2015

Example to case #1: Marinexplore.org 40,000+ data streams from 33 organizations of open data

Advanced Data Discovery Raster data / heat map overlays Access with third party applications

Raster data / quiver plots Graph Monitor Build custom datasets

7 February 2015

Consolidate dataflows, organize and make sense of your data

Access your data with 3rd party tools and systems

One interface to search and discover your local and remote data

Securely exchange, acquire or sell datasets

Build advanced domain specific solutions without hassle

Industrial IoT Platform for Real-World Sensor Networks designed for ocean, land, air and space data

8 February 2015

Integrate, curate, control access and monitor dataflows

Analytics dashboard Integrate and configure new data channels

Roles, authorizations and logs Dataflow dashboard

9 February 2015

Case #2: Agriculture + data science Models, predictions, outliers, multi-source data

New companies are emerging:

• Predict a farmer’s profit statement based using adaptive models

(drone data + environmental data + in-situ data + soil/seed/pesticide choices)

• Open data is sparse and scattered

• Data dissemination is hard

• Farmer’s data input in hard

• “Do we really need to build a data platform to build our business?”

What if newco’s and farmers could focus on what they know best?

10 February 2015

Case #3: Surveillance and tactical in-flight data Having a 2-way data aggregation is critical

• End-users using a variety of drones need data versatility

• “How can I get a data-edge in a chase scenario?”

• Situational awareness based on the same surrounding data

for all unmanned vehicles in your fleet

• Drones need more accurate surrounding data than manned

crafts (don’t fly with your windows taped)

11 February 2015

Case #4: Planet OS data exchange in a swarm scenario Secure data exchange for tactical navigation & interoperability

• Connected cars / drones / UGVs / AUVs / fleets

• Multi-particle computation for ensuring privacy

• Collision avoidance using encrypted data

• Behavioral data collection using encrypted data

12 February 2015

Main Use Cases

Visualisation

Analytics

Event Processing

Models

Curation & Logistics

Marketplace

Search

ExplorationTECHNICAL DATA

• Spatio-Temporal • Sensors • Observations • Models • Predictions • Metadata

13 February 2015

Interfaces and IntegrationsDATA FORMATS

• CSV, Spreadsheet • GeoTiff, ESRI Raster • HDF, NetCDF (CF), GRIB • Shapefile, KML • OGC Geopackage • Metadata: Exif, SRID, ISO 19139 • Images, Audio, Video • Custom formats

DATA ACCESS / EXPORT

• CSV, Spreadsheet • GeoTIFF, HDF, NetCDF (CF) • OGC WFS / WCS / WMS • OpenDAP • ESRI ArcGIS • ISO-19139 • Custom formats and protocols

DATA SOURCES

• HTTP, FTP • NFS, Samba, Sharepoint • OpenDAP, THREDDS, ERDAP • OGC WFS / WCS / WMS • ESRI ArcGIS, PostGIS • Archives: ZIP, etc • Custom protocols

14

Index the real world