Real-time data analysis with Azure Stream Analytics

Real-time data analysis with Azure Stream Analytics

Michael Johnson

What is streaming data?

Batch data Streaming data

Where do we find streaming data

What is Azure Stream Analytics and why would you want to use it

• What• Real-time event processing engine that helps to unlock deep insights from

data.

• Why• Easy to use

• Scalable

• Reliable, Repeatable and quick to recover

• Low cost

Azure stream analytics architectureAzure stream analytics

Inputs Outputs

Input source

• Streaming Data• Event hubs

• Blob storage

• IOT Events hubs

• Reference data• Blob storage

Output sinks

Azure data lake

Power BI

Azure SQL database Azure document DB Azure Blob storage

Azure Service bus

Azure Event hubAzure table storage

Query language overview

• DML Statements• SELECT• FROM• WHERE• GROUP BY• HAVING• CASE• JOIN• UNION

• Date and time• DATEPART

• DAY

• MONTH

• YEAR

• DATEDIFF

• DATEADD

• Aggregation functions• SUM

• COUNT

• AVG

• MIN

• MAX

• Scaling functions• WITH

• PARTITION BY

• Windowing functions• WITH

• PARTITION BY

• String functions• LEN

• CONCAT

• CHARINDEX

What does a query look like?

Tumbling windowsTUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] )

Hopping windowHOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] )

Sliding windowSLIDINGWINDOW ( timeunit , windowsize )

Javascript UDF intergration

• Supports stateless, compute only functions

• No external connectivity

• Why• REGEX

• Math libraries

Machine learning integration

Geospatial Queries (In public preview)• Geospatial Functions

• types• CreatPoint

• CreatePolygon

• CreateLineString

• Functions• ST_Distance

• ST_WITHIN

• ST_OVERLAP

• ST_INTERSECTS

Deployment options

Azure portal .Net SDK PowerShell

Pricing

• Streaming unit• 1 streaming unit = 1MB/Second

• 1 x 60 x 60 x 24 = 84.375 GB/day

• Data volume + streaming unit cost• Data volume = $ 0.001 or R 0.014 per GB

• Streaming unit = $ 0.12 or R1.656 per hour

• If run at max through put for a month• Total data ingested = 2.47 TB

• Data = R 35.44

• Streaming Unit = R1 192.32

• Total cost = R 1 227.76

DEMO

Questions