The Power of Now! Azure Stream Analytics - Microsoft ITPro AirLift

22 de Janeiro \\ Microsoft Lisbon Experience

The Power of Now!

Rui Quintino | DevScope

22 Jan 2015

[email protected]

rquintino.wordpress.com

twitter.com/rquintino

Azure Stream Analytics

The traditional data warehouse

Modern data warehouse

What are customers wanting to do?

Canonical scenarios for real-time processing

KUKA Systems Group created an IoT-powered plant that

centrally connects and monitors hundreds of robots.

Introducing Azure Stream Analytics

Procure HW Infrastructure and setup

Code for ingress, processing and egress

Plan for resiliency, such as HW failures

Design solution

Build Monitoring and Troubleshooting

• Built-in monitoring

– View your system’s performance

at a glance

– Help you find the cost-optimal

way of deployment

3 lines of code in Stream Analytics

Thousand lines of code in other solutions

• Only SQL queries needed

– Developers uses declarative

SQL commands

– Some functions take several

lines of code versus thousands

from other solutions

• Implement temporal functions

• Manage out-of-order events

• Manage actions on late events

End-to-end stream processing architecture

Sample Scenario – Toll Station

TollId EntryTimeLicense

PlateState Make Model Type Weight

1 2014-10-25T19:33:30.0000000Z JNB 7001 NY Honda CRV 1 3010

1 2014-10-25T19:33:31.0000000Z YXZ 1001 NY Toyota Camry 2 3020

3 2014-10-25T19:33:32.0000000Z ABC 1004 CT Ford Taurus 2 3800

2 2014-10-25T19:33:33.0000000Z XYZ 1003 CT Toyota Corolla 2 2900

1 2014-10-25T19:33:34.0000000Z BNJ 1007 NY Honda CRV 1 3400

2 2014-10-25T19:33:35.0000000Z CDE 1007 NJ Toyota 4x4 1 3800

… … … … … … … …

EntryStream - Data about vehicles entering toll stations

TollId ExitTime LicensePlate

1 2014-10-25T19:33:40.0000000Z JNB 7001

1 2014-10-25T19:33:41.0000000Z YXZ 1001

3 2014-10-25T19:33:42.0000000Z ABC 1004

2 2014-10-25T19:33:43.0000000Z XYZ 1003

… … …

ExitStream - Data about cars leaving toll stations

LicensePlate RegistartionId Expired

SVT 6023 285429838 1

XLZ 3463 362715656 0

QMZ 1273 876133137 1

RIV 8632 992711956 0

… … ….

ReferenceData - Commercial vehicle registration data

Application Components

Components of an Azure Stream Analytics Application

Azure SQL DB

Azure Event Hubs

Azure Blob StorageAzure Blob Storage

Azure Event Hubs

Reference Data

Query runs continuously against incoming stream of events

Events

Have a defined schema and

are temporal (sequenced in

time)

Temporal Windows• Tumbling Windows

– Repeating, non-overlapping, fixed interval windows

• Hopping Windows– Generic window, overlapping, fixed size

• Sliding Windows– Slides by an epsilon and produces output at the occurrence of an event

Tumbling Window

SELECT System.TimeStamp AS OutTime, TollId, COUNT (*) FROM Input TIMESTAMP BY EntryTimeGROUP BY TollId, TumblingWindow(minute,5)

Hopping Windows

SELECT System.TimeStamp AS OutTime, TollId, COUNT (*)FROM Input TIMESTAMP BY EntryTimeGROUP BY TollId, HoppingWindow(minute, 10 , 5)

Sliding Windows

SELECT System.TimeStamp AS OutTime, TollId, COUNT (*)FROM Input TIMESTAMP BY EntryTimeGROUP BY TollId, SlidingWindow(minute, 3)HAVING Count(*) > 3

Finds all toll booths which have served more than 3 vehicle in the last 3 minutes

Explore the latest in SQL ServerVisit www.microsoft.com/sql to learn more about our latest innovations

Evaluate SQL Server 2014Get hands on – visit Microsoft’s TechNet evaluation center and

evaluate SQL Server 2014 today!

Evaluate Windows Server 2012 R2Get hands on – visit Microsoft’s TechNet evaluation center and

evaluate Windows Server 2102 R2 today!Microsoft Azure