IoT Architecture - are traditional architectures good enough or do we need new approaches?

2014 © Trivadis

19.11.2014 IoT Architektur

1

IoT Architektur Sind traditionelle Architekturen gut genug, oder braucht es neue Ansätze?

Bernd Trops (Talend) Guido Schmutz (Trivadis) 24.03.2015

Open Source Integration

2014 © Trivadis

Bernd Trops

•  Working for Talend for more than 7 years •  Co-Author of different books •  Consultant, Trainer and Software Architect for Java, SOA •  Masons of SOA: http://www.soacommunity.com/index.php/institutional/masons-of-soa •  Director Professional Services DACH at Talend •  More than 25 years of software development

experience

•  Contact: [email protected]

19.11.2014

IoT Architektur 2

2014 © Trivadis

Guido Schmutz

•  Working for Trivadis for more than 17 years

•  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and

Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis

•  More than 25 years of software development experience

•  Contact: [email protected] •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz


3

2014 © Trivadis

Agenda

1.  Introduction / Motivation

2.  IoT Reference Architecture

3.  Components supporting the IoT Reference Architecture

4.  Talend – Unified Platform for IoT and Big Data

5.  Summary


4

2014 © Trivadis

Data, Data, Data – Predicted Growth from IoT


5

Source: Brideworks http://bit.ly/19Msx5N

2014 © Trivadis

Data as an Asset - Store Anything?

But then data is just too valuable to delete! We must store anything!

Nonsense! Just store the data

you know you need today!

It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today!

6


2014 © Trivadis

Requirements for an IoT Architecture

•  Support for failure •  systems should gracefully degrade into lower performing states •  If a failed component recovers, it should be able to rejoin the cluster

•  Recoverability •  In the event of a failure, no data should be lost

•  Consistency •  Failure of one task should not affect the result of the entire job

•  Scalability •  Adding load leads to a decline of performance, not failure •  Increasing resources results in proportional increase in capacity •  Scale-up but also scale-down => elasticity

7


2014 © Trivadis

Agenda





5.  Summary


8

2014 © Trivadis

IoT Solution Types


9

Device

On-Device Logic

Gateway

Processing Backend

Enterprise Apps

Enterprise IoT M2M Local Hub Device to Device (P2P)

Device to Backend

Adapted from: http://enterprise-iot.org

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)

Enterprise Applications

WS

External Cloud Service

Providers

BPM and SOA Platform

Event

Business Logic/Rules

Business Intelligence

IoT Device

Services

WS

Event

Processes

Gateway

Event Procesing

IoT Smart Device

Viszualization

Analytics

DB


10

API G

atew

ay

= one way = request/response

1) IoT with Simple Event Processing

10

Mobile Apps

DB

Rich (Web) Client Apps

DB

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)

Stream Processing Infrastructure


WS


Providers


Event



IoT Device

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP

ESP/CEP

IoT Smart Device

Viszualization

Analytics

DB

DB

DB


11

API G

atew

ay


11

Mobile Apps

DB


DB

Social Media

Streams

2) IoT with Event Stream/Complex Event Processing

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)

3) IoT with Big Data Analytics


WS


Providers


Event



IoT Device

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/ CEP

Hadoop Big Data Infrastructure HDFS

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB DB


12

API G

atew

ay


12

Mobile Apps

DB


DB

Social Media

Streams

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)



WS


Providers


Event



IoT Device

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB


13

API G

atew

ay

4) IoT with “Lambda” Architecture


13

Mobile Apps

DB


DB

Social Media

Streams

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)



WS


Providers


Event



IoT Device

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Replay to reprocess

IoT Smart Device

Viszualization

Analytics

DB

DB

DB


14

API G

atew

ay

5) IoT with “Kappa” Architecture


14

Mobile Apps

DB


DB

Social Media

Streams

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)



WS


Providers


Event



IoT Device

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB


15

API G

atew

ay

6) IoT with Big Data Advanced Analytics


15

Mobile Apps

DB


DB

Social Media

Streams

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)


IoT Reference Architecture


WS


Providers


Event



IoT Device

16

Mobile Apps

DB


DB

Social Media

Streams

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB


16

API G

atew

ay


2014 © Trivadis

IoT Reference Architecture


17 En

terp

rise

Serv

ice

Bus

(ESB

)



WS


Providers


Event



IoT Device

17

Mobile Apps

DB


DB

Social Media

Streams

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB

API G

atew

ay


2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)



WS


Providers


Event



IoT Device

18

Mobile Apps

DB


DB

Social Media

Streams

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB

API G

atew

ay


18

IoT Reference Architecture – Supporting Components

2014 © Trivadis

Agenda





5.  Summary


19

2014 © Trivadis

Enterprise Event Bus - Apache Kafka

•  A distributed publish-subscribe messaging system

•  Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)

•  Initially developed at LinkedIn, now part of Apache

•  Does not follow JMS Standards and does not use JMS API

•  Kafka maintains feeds of messages in topics


20

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

0 1 2 3 4 5 6 7 8 9 1 0

1 1

1 2

0 1 2 3 4 5 6 7 8 9

0 1 2 3 4 5 6 7 8 9 1 0

1 1

1 2

Anatomy of a topic:

Partition 0

Partition 1

Partition 2

Writes

old new

2014 © Trivadis

Enterprise Event Bus - Apache Kafka

Kafka at LinkedIn

Up to 2 million writes/sec on 3 cheap machines §  Using 3 producers on 3 different machines


21

10+ billion writes per day

172k messages per second

(average)

55+ billion messages per day

to real-time consumers

http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

2014 © Trivadis

Management /Monitoring

Infrastructure

Batch (Big) Data Processing - Hadoop Ecosystem

22

Core

Analytics Workflow/Job Unstructured Data Sources

Structured Data Sources

SQL on Hadoop

Serialization Data Storage Security


2014 © Trivadis

Batch (Big) Data Processing - Classical Hadoop vs. Spark

Spark Hadoop Map Reduce


23

HDFS

Map

Map

Reduce

Reduce

Data Data

Map

Join

Transform

Cache

Filter

Join

Data

2014 © Trivadis

Stream Processing – Apache Storm

Tuple •  Immutable Set of Key/value pairs

Stream •  an unbounded sequence of tuples that can be processed in parallel by Storm

Topology •  Wires data and functions via a DAG (directed acyclic graph) •  Executes on many machines similar to a MR job in Hadoop

Spout •  Source of data streams (tuples) •  can be run in “reliable” and “unreliable” mode

Bolt •  Consumes 1+ streams and produces new streams •  Complex operations often require multiple

steps and thus multiple bolts


24

Spout

Spout

Bolt

Bolt

Bolt

Bolt

Source of Stream B

Subscribes: A Emits: C

Subscribes: A Emits: D

Subscribes: A & B Emits: -

Subscribes: C & D Emits: -

T T T T T T T T

2014 © Trivadis

Data Storage (Batch) – Cloudera Impala

High-performance SQL engine for vast amounts of data

•  Massively-parallel processing (MPP)

•  Inspired by Google's Dremel project

•  Query latency measured in milliseconds

Impala runs on Hadoop clusters

•  Can query data stored in HDFS or HBase tables

•  Reads and writes data in common Hadoop file formats

Developed by Cloudera

•  100% open source, release under the Apache software license


25

2014 © Trivadis

Data Storage (Speed) – Apache Cassandra

•  Developed at Facebook

•  Open source distributed database management system

•  Professional grade support from company called DataStax

•  Main Features §  Real-Time §  Highly Distributed §  Support for Multiple Data Center §  Highly Scalable §  No Single Point of Failure §  Fault Tolerant §  Tunable Consistency §  Cassandra Query Language (CQL)


26

2014 © Trivadis

Data Storage (Speed) – Apache Cassandra

•  Don’t think of relational table => more of a sorted map

•  Know your application => model around the queries

•  De-normalize and duplicate for read performance

•  Index is not an afterthought, anymore=> index upfront

•  Think of physical storage structure


27

2013-08 Day-1, keyword-1=>100

Day-2, keyword-1=>150

Day-3, keyword-1=>170 …. Day-31,

keyword-1 =>170

2013-08-31 Hour-1, keyword-1 =>10

Hour-2, keyword-1 =>15

Hour-3, keyword-1 =>17 …. Hour-24,

keyword-1 =>17

2013-08-31-10 Minute-1, keyword-1=>2

Minute-2, keyword-1=>3

Minute-3, keyword-1 =>5 …. Minute-60,

keyword-1=>2

Row-key Columns à

2014 © Trivadis

Agenda





5.  Summary


28

2014 © Trivadis

Ente

rpris

e Se

rvic

e Bu

s (E

SB)



WS


Providers


Event



IoT Device

29

Mobile Apps

DB


DB

Social Media

Streams

Ente

rpris

e Ev

ent B

us (I

ngre

ss)

Ente

rpris

e Ev

ent B

us

Services

WS

Event

Processes

Gateway

ESP/CEP


ESP/CEP

Batch Processing

IoT Smart Device

Viszualization

Analytics

DB

DB

DB

DB

API G

atew

ay

Talend – Unified Platform for IoT and Big Data


29

2014 © Trivadis

Talend Kafka Support


30

2014 © Trivadis

Talend Spark Support


31

2014 © Trivadis

Agenda





5.  Summary


32

2014 © Trivadis

Summary

•  IoT – The technologies are ready to use

•  IoT tends to make Big Data infrastructures necessary

•  There are architectures to deal with IoT -> Backend

•  Know your use case/requirements to choose the right architecture

•  Can my existing backend landscape handle the new IoT load?

•  Do I have to handle huge amount of events in “real-time”?

•  Do I need to filter/aggregate data before invoking existing backend systems?

•  Do I want to do Advanced Analytics (predictive analytics) where large amounts of historical information is necessary?

•  Business cases starting to increase


33

2014 © Trivadis

Weitere Informationen...


34

INFOBOX – Lesen und Löschen •  Folie wenn auf weitere Informationen

verwiesen werden soll, also z.B. Bücher, Websiten, etc.

2014 © Trivadis

BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

Fragen und Antworten...

2013 © Trivadis

Guido Schmutz

Technology Manager

[email protected]


INFOBOX – Lesen und Löschen •  Die Schlussfolie steht in zwei Varianten

zur Verfügung, einmal für die Kontaktdaten eines Referenten, einmal in der Variante für zwei oder mehr Referenten

•  Name, Titel und Location jeweils untereinander in eine Zeile (Shift+Return)

•  Die Idee ist das diese Folie als letzte Folie (auch für Fragen und Antworten) am Ende der Präsentation lange stehen bleibt, somit haben die Zuhörer die Möglichkeit die Kontaktdaten aufzuschreiben J

IoT Architecture - are traditional architectures good enough or do we need new approaches?

Internet

trivadis data

iot reference architecture

iot architecture support

trivadis iot solution

trivadis agenda

trivadis requirements

big data technologies

trivadis guido schmutz