Holistic, Distributed Stream Processing in IoT Environments 23. 03. ‘18 UK Systems Research Workshop Peter Michalák CDT Cloud Computing for Big Data Prof. Paul Watson School of Computing Prof. Mike Trenell Medical School Dr. Sarah Heaps School of Maths and Stats
17
Embed
Holistic, Distributed Stream Processing in IoTEnvironmentssysws.org.uk/workshop/2018/54-michalak-streams.pdf · Step count algorithm[1]in EPL. PATHfinder: ... Bat Life (h) pp03 Ω1
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Holistic, Distributed Stream Processing in IoT Environments
23. 03. ‘18 UK Systems Research Workshop
Peter MichalákCDT Cloud Computing
for Big Data
Prof. Paul WatsonSchool of Computing
Prof. Mike TrenellMedical School
Dr. Sarah HeapsSchool of Maths
and Stats
Stream Processing in IoT
Telemetry
Sensors Field Gateways
MessageBroker
Stream Processing Engine
Clouds
1
Telemetry
Command and ControlNotification
Sensors Field Gateways
MessageBroker
Stream Processing Engine
Clouds
DB
Holistic Stream Processing in IoT
2
Healthcare use case Behavioural Prompts & Feedback
BehaviouralPrompt
Feedback
3
ESPer/Spark/Storm
sky is the limit
C / JavaScript
Objective C / Swift
Java / Kotlin ..
Challenges
• Energy
• Pebble Watch battery life ~7 days
• Streaming raw accelerometer data reduces battery life ~18 hours
• Hand-crafting bespoke solutions
• Definition of data stream processing
• Programming for Heterogeneous Platforms
• Multitude of devices, APIs and programming languages
PATHfinder : High-Level Declarative Description of Computation
Non-functional Requirements
Description of the ComputationIn
put Resource
Catalogue
• Event Processing Language (EPL) from Esper
• High Level Declarative Description of Computation
• SQL based with extended grammar to support CEP operations
• Decomposable into directed graph of stream operators
6
1. INSERT INTO AccelEventSELECT getAccelData(25, 60) FROM AccelEventSource
2. INSERT INTO EdEventSELECT Math.pow(x*x+y*y+z*z, 0.5) AS ed, tsFROM AccelEvent WHERE vibe=0
3. INSERT INTO StepEventSELECT ed1(’ts’) as ts FROM EdEventMATCH RECOGNIZE (MEASURES A AS ed1, B AS ed2 PATTERN (A B) DEFINE A AS (A.ed > THR), B AS (B.ed ≤ THR))
4. INSERT INTO StepCount SELECT count(*) as steps FROM StepEvent.win:time_batch(120 sec)
5. SELECT persistResult(steps, "time_series", "step_sum") FROM StepCount
[1] N. Zhao, “Full-featured pedometer design realized with 3-axis digital accelerometer,” Analog Dialogue, vol. 44, no. 06, 2010.
Step count algorithm[1] in EPL
PATHfinder : High-Level Declarative Description of Computation
1. INSERT INTO AccelEventSELECT getAccelData(25, 60) FROM AccelEventSource
2. INSERT INTO EdEventSELECT Math.pow(x*x+y*y+z*z, 0.5) AS ed, tsFROM AccelEvent WHERE vibe=0
3. INSERT INTO StepEventSELECT ed1(’ts’) as ts FROM EdEventMATCH RECOGNIZE (MEASURES A AS ed1, B AS ed2 PATTERN (A B) DEFINE A AS (A.ed > THR), B AS (B.ed ≤ THR))
4. INSERT INTO StepCount SELECT count(*) as steps FROM StepEvent.win:time_batch(120 sec)
5. SELECT persistResult(steps, "time_series", "step_sum") FROM StepCount
[1] N. Zhao, “Full-featured pedometer design realized with 3-axis digital accelerometer,” Analog Dialogue, vol. 44, no. 06, 2010.
Step count algorithm[1] in EPL
Non-functional Requirements
Description of the ComputationIn
put
Logical Plan
Resource Catalogue
• Logical Optimisation[2]
• Pushing Selects &Windows closer to the data source
• Physical Optimisation• Enumerating Physical Plans• Placement of the physical plans
on available infrastructure• 225 Physical Plans
• Physical Plan Pruning• Removing non-deployable plans
based on infrastructure capabilities
• 18 Physical Plans
Physical Plan
8
PATHfinder: Optimisation
[2] Hirzel, Martin, Robert Soulé, Scott Schneider, Buğra Gedik, and Robert Grimm. ‘A Catalog of Stream Processing Optimizations’. ACM Computing Surveys 46
PP0 PP1 PP2 PP3 PP4 PP5
PP6 PP7 PP8 PP9 PP10 PP11
PP12 PP13 PP14 PP15 PP16 PP17
PP18 PP19 PP20 PP21 … PP225
x xx
xx
x x
xxxx
x
Non-functional Requirements
Description of the ComputationIn
put
Logical Plan
Resource Catalogue
• Cost Models• Energy cost model[3]
• Power coefficients with confidence Intervals• Estimated Battery LifePhysical Plan
Cost Models
9
PATHfinder: Energy Cost Model
[3] M. Forshaw, N. Thomas, and A. S. McGough, “The case for energy- aware simulation and modelling of internet of things (iot),” ACM ENERGY-SIM, 2016.
Non-functional Requirements
Description of the ComputationIn
put
Logical Plan
Resource Catalogue
• Cost Models• Energy cost model• Power coefficients with
confidence Intervals• Estimated Battery LifePhysical Plan
Cost Models
10
PATHfinder : Energy Cost Model
Non-functional Requirements
Description of the ComputationIn
put
Logical Plan
Resource Catalogue
• Cost Models• Energy cost model• Power coefficients with
confidence Intervals• Estimated Battery LifePhysical Plan
Cost Models
11
PATHfinder : Energy Cost Model
Non-functional Requirements
Description of the ComputationIn
put
Logical Plan
Resource Catalogue
• Cost Models• Energy cost model• Power coefficients with
confidence Intervals• Estimated Battery LifePhysical Plan