How to get Real-Time Value from your IoT Data - Datastax

Confidential 1

How to get Real-Time Value from your IoT Data

Vincent Poncet Solution Engineer EMEA

IoT Application Characteristics

Real-Time DistributedAlways-OnContextual Scalable

Platform for IoT Applications

DataStax is a registered trademark of DataStax, Inc. and its subsidiaries in the United States

and/or other countries.4

From validation to momentum.

400+Employees

$190MFunding

500+Customers

Founded in April 2010Santa Clara • San Francisco • Austin •

London • Paris • Berlin • Tokyo • Sydney

(Series E – Sept. 2014) 30% +

2016 World’s Best

100 Cloud Companies

Ranked #1 in multiple operational

database categories

THE CHALLENGE: Collect sensor data from millions of

devices from around the world and

manage trillions of transactions per day

• GE offers the first Industrial Cloud Platform called

GE Predix. Datastax is part of the data services

layer within the platform

• DSE will collect sensor data from millions of

devices from around the world to help GE provide

predictive maintenance to their customers and

increase operational efficiencies

• Predix manages trillions of transactions

per day. DSE was recognized as the only

solution that could support this scale and

data center replication

First Utility

• First Utility offers a disruptive, modern

application called My Energy that gives

customers total transparency to understand &

manage their energy consumption

• Each Smart Meter produces up to 17,000

readings per year

• DSE provides the distributed, responsive &

intelligent foundation to power My Energy at

• As a result, customers use 5-6% less energy

and further reduce their energy bills

THE CHALLENGE: Drive better customer experiences by

giving customers the information they

need to control their energy usage

through Smart Meters technology

Traxens

THE CHALLENGE: Implement a solution for global,

real-time, end-to-end monitoring of

containers door to door and proactive

alerts for issues

• Traxens offers an IOT service, Trax-Hub, for

real-time, end to end global monitoring of

containers door to door

• Alerts for open boxes, temperature changes,

• Granular monitoring of individual

containers: Traxens can store information

on all containers (up to 20,000 in one ship),

and hundreds of attributes per container

• Scalable platform for future needs

What is Apache Cassandra?

Apache Cassandra

Do not distribute without

consent.

• Distributed NoSQL Database

• Google Big Table

• Amazon Dynamo

• Continuous Availability

• Disaster Avoidance

• Linear Scale Performance

• Add nodes to scale

• Runs on Commodity Hardware

• Cloud or on Premise

Francisco

New York

Munich

Apache Cassandra Disaster Avoidance

Francisco

New York

Munich

Example Data Model

Sensor collects data

Cassandra stores in sequence

Application reads in sequence

Car Sensor Use Case

• Store data per sensor

• Store time series in order: first to last

• Get all data for one sensor

• Get data for a single date and time

• Get data for a range of dates and times

Needed Queries

Data Model to support queries

Use Case

Sensor Id and Time are unique

Store as many as needed

CREATE TABLE car_stats (

sensor_id text,

collect_time timestamp,

temperature text,

longitude text,

latitude text,

speed text,

PRIMARY KEY (sensor_id,collect_time)

INSERT INTO car_stats(sensor_id,collect_time,temperature,longitude,latitude,speed)

VALUES ('1234ABCD','2013-04-03 07:01:00','19C',’134.231’,‘234.234’,’60kmh’);

VALUES ('1234ABCD','2013-04-03 07:02:00','20C',’135.230’,‘237.239’,’65kmh’);

VALUES ('1234ABCD','2013-04-03 07:03:00','20C',’137.431’,‘240.793’,’68kmh’);

VALUES ('1234ABCD','2013-04-03 07:04:00','21C',’138.589’,‘234.234’,’69kmh’);

Data Model

SELECT sensor_id,collect_time,temperature,longitude,latitude,speed

FROM car_stats

WHERE sensor_id='1234ABCD';

Sensor_id Collect_time temperature longitude latitude Speed

1234ABCD 2015-04-03

07:01:00

19C 134.231 234.234 60kmh

1234ABCD 2015-04-03

07:02:00

20C 135.230 237.239 65kmh

1234ABCD 2015-04-03

07:03:00

20C 137.431 240.793 68kmh

1234ABCD 2015-04-03

07:04:00

21C 138.589 234.234 69kmh

Storage Model – Logical View

FROM car_stats

WHERE sensor_id='1234ABCD';

Merged, Sorted and Stored Sequentially

2015-04-03 07:01:00 2015-04-03 07:02:00 2015-04-03 07:03:00 2015-04-03 07:04:00

19C 134.2

20C 135.2

65kmh 20C 137.4

68kmh 21C 138.5

Storage Model – Disk Layout

Range queries

“Slice” operation on disk

FROM car_stats

WHERE sensor_id='1234ABCD'

AND collect_time >= '2015-04-03 07:01:00'

AND collect_time <= '2015-04-03 07:04:00';

Single seek on disk

Query Patterns

2015-04-03 07:01:00 2015-04-03 07:02:00 2015-04-03 07:03:00 2015-04-03 07:04:00

19C 134.2

20C 135.2

65kmh 20C 137.4

68kmh 21C 138.5

Range queries

“Slice” operation on disk

Sorted by

collect_time

FROM car_stats

WHERE sensor_id='1234ABCD'

AND collect_time >= '2015-04-03 07:01:00'

AND collect_time <= '2015-04-03 07:04:00';

Query Patterns

Sensor_id Collect_time temperature longitude latitude Speed

1234ABCD 2015-04-03

07:01:00

19C 134.231 234.234 60kmh

1234ABCD 2015-04-03

07:02:00

20C 135.230 237.239 65kmh

1234ABCD 2015-04-03

07:03:00

20C 137.431 240.793 68kmh

1234ABCD 2015-04-03

07:04:00

21C 138.589 234.234 69kmh

Cassandra Data Modeling

Requires a different mindset than RDBMS modeling

Know your data and your queries up front

Queries drive a lot of the modeling decisions (i.e. “table per query” pattern)

Denormalize/Duplicate data at write time to do as few queries as possible

come read time

Remember, storage is cheap and writes in Cassandra are FAST ( about

1,000 inserts / second per physical CPU core )

DataStax Enterprise DSE

I’ve ingested my data, now what?

Platform for IoT Applications

DataStax is a registered trademark of DataStax, Inc. and its

subsidiaries in the United States and/or other countries.

DataStax EnterpriseCompany Confidential

Offline

Application

External

Spark or

Hadoop

Cluster

Spark/

Hadoop

Real Time

Analytics

Search

Certified Apache Cassandra

No Single Point of Failure | Linear Scalability | Always-On

DSE – Fully Integrated Technology Stack

Ease of

DataStax

Studio

OpsCenter

Services

Monitoring,

Operations

Latency

Memory

DatabaseOperational

Resiliency

System

Advanced

SecurityDSEFS

Analytics

Transformations

• Ready and certified for production environments.

• Rigorous certification process:

• Extensive quality assurance testing.

• Performance and scale tests with 1,000 node clusters.

• 3rd party software validation.

• Certified for key supported platforms.

DataStax Enterprise – Certified Cassandra

• Embedded Spark

• ETL workloads, Real-Time Streaming Analytics, SQL Operational Analytics on

Cassandra.

• DSE benefits:

• Spark Master HA

• Integrated security

• Support

DSE Analytics

HTTP Application Message Queue

Streaming

Analytics

Near Real Time

Analytics

Real-time

DSE Multi-Workload Analytics Architecture

• DSE Search inherits all the power and capabilities of Solr and builds on top of it

to create even more powerful enterprise search functionality

• Built-in scale out and continuous availability and multiple data centers support

• Automatic indexing when inserting and updating in Cassandra

• Search Capabilities integrated into Cassandra Query Language

• Multi-criteria

• Full text

• Geospatial

• Faceting

• Auto-completion

DSE Search

HTTP Application Message Queue

Streaming

Analytics

Near Real Time

Analytics

Real-timeSearch

DSE Multi-Workload Analytics Architecture

• Allows one-way replication from “edge” cluster to another,

centralized hub cluster.

• Ideal for retail, energy, and other “edge of the internet of things” use

cases.

• Hub and spoke

DSE Advanced Replication

• Able to automatically move data to different storage media based on

defined criteria.

• Helps reduce storage costs by relegating lesser-used or older data to

less expensive storage devices.

• Works on a granular per-row basis.

DSE Tiered Storage

Confidential 32

• Distributed file system, masterless, API

compatible with HDFS

• Resiliency of metadata, being stored in

Cassandra tables

• Cost effective, cold storage of data

• Staging

• Archiving

• Analytics (with Spark)

Confidential 33

Storage Temperature Management

• Business value of data record per byte is low in IoT use cases

• Being able to optimize the cost of storage depending on the usage of the data is

• A tiering / temperature management approach is a relevant response

• Hot Fast storage using SSD for fresh data

• Warm cost effective storage using HDD for older data (in-DB online archive)

• Cold cheapest storage using file system for long term data (out-DB archive)

• Can be used with Spark for analytical usages

Hot Data

Tiered Storage

Warm Data

Tiered Storage

Cold Data

• Transparent Data Encryption of ALL DSE data at rest

• Role based access control

• Unified authentication: Allows multiple security authentication protocols (e.g.

Kerberos, LDAP, Active Directory, internal Cassandra) to be used on the same

database cluster.

• Data Auditing

DSE Enterprise Security

Build and Manage

Interact with DataStax

Enterprise from your

application

Create objects via DDL (e.g.

CREATE…)

GRANT/REVOKE

INSERT, UPDATE, DELETE

Query data with SELECT

Certified DataStax drivers:

Community drivers:

Java C# Python C++

Node.js ODBC PHP Ruby

Closure Erlang Haskell Rust

The Cassandra Query

Language (CQL)

Explore, query, and

analyze DSE

• Visually Create and

Navigate Database

Objects via CQL

• Gremlin Query Language

Support

• Auto-completion, result set

visualization, execution

management, and much

• Friendly Fluent APIDataStax is a registered trademark of DataStax, Inc. and its subsidiaries in the United States

Studio

Visual management for

• Automate what no one

likes – backups, repairs

• REST API to work in your

• Instantly manage your

cluster, scaling up or down

at a moment’s notice

• Monitor your cluster and

follow best practices,

ensuring a secure

environmentDataStax is a registered trademark of DataStax, Inc. and its subsidiaries in the United States

OpsCenter

• Designed to automatically handle many maintenance and

management tasks.

• Makes DSE easy to work with.

• Services included:

• Repair service

• Capacity service

• Performance service

• Best Practice service

• Backup/Restore service

DataStax Automatic Management Services

• 24x7x365

• Production and non-production environments.

• Health checks for assistance on architecture,

design, and tuning.

• Certified service packs

• Hot-fix support and back porting of bug fixes

DataStax Expert Support

DataStax Managed Cloud

• DSE on AWS with Managed Provisioning

and Scaling by DataStax

• 24x7x365 Coverage,

Lights-Out Management

• System Configuration and Tuning to Meet

Customer Specific Requirements

• Architecture Advisory Services, Guidance

and Best Practices

A Fully Managed, Secure Architecture

Cassandra, Apache, Spark, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

Thank you

We are the powerbehind the moment.

Confidential

How to get Real-Time Value from your IoT Data - Datastax

Technology

DataStax et Apache Cassandra pour la gestion des flux IoT

Business value Drivers for IoT Solutions

DataStax Java Driver Walkthrough

Click-IoT - A platform for getting value from your IoT Big.....

Scaling DataStax in Docker

Extending Value Creation Through IoT, Processes … Value...

Develop Scalable Applications with DataStax Drivers (Alex...

Creating, Capturing and Sharing Value from the IoT ·...

Hello DataStax Enterprise Graph

What is DataStax Enterprise?

IoT 2014 Value Creation Workshop: SDIL

DataStax Enterprise - Cas d'usage

Microsoft Azure IoT services Reference...

IoT Value Roadmap

Best Practices in Deploying and Managing DataStax...

DataStax | Graph Data Modeling in DataStax Enterprise (Artem...