Business Intelligence in Azure - SQL Relay in...Business Intelligence in Azure Alex Whittles Alex Whittles Business Intelligence Consultancy Data Modelling ETL Systems Data Warehousing

[email protected]

PurpleFrogSystems.com

PurpleFrogSystems.com/blog

@PurpleFrogSys

Business Intelligence in Azure

Alex Whittles

Alex Whittles

Business Intelligence ConsultancyData Modelling ETL SystemsData Warehousing Reporting SystemsOLAP Cubes Managed Service

• SQL Relay Committee SQLRelay.co.uk

• SQL Bits Committee SQLBits.com

• Birmingham SQL UG SQLMidlands.com

• Birmingham Azure UG AzureBirmingham.uk

• MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD

• Run BI Consultancy

Alex Whittles

Business Intelligence ConsultancyData Modelling ETL SystemsData Warehousing Reporting SystemsOLAP Cubes Managed Service

• SQL Relay Committee SQLRelay.co.uk

• SQL Bits Committee SQLBits.com

• Birmingham SQL UG SQLMidlands.com

• Birmingham Azure UG AzureBirmingham.uk

• MSc in Business Intelligence, CEng, CITP, FBCS, FIOEE, MIET, MIOD

• Run BI Consultancy

Cortana Analytics Data Flow

Business Intelligence in Azure

Options:1. Migrating on-premise BI solutions to Azure2. Batch load BI systems in Azure3. Real-time BI systems in Azure

For each:Staging / ETL / Data Warehouse / Cube

On-Premise Batch Load BI

SQL ServerStaging Database

SQL ServerData Warehouse

SSASCube

Source Data

SSISETL

Azure IAAS Solution

SSISETL

Source Data

SSISETLSQL Server

Data Warehouse

SSASCube

[Infrastructure As A Service]

Azure Virtual Machine

SQL ServerStaging Database

On-Premise

Better Azure IAAS Solution

Source Data

SSISETL SQL Server

Data Warehouse

SSASCube

[Infrastructure As A Service]

Azure Virtual Machine

SQL ServerStaging Database

On-Premise

VPN GatewayOr ExpressRoute

PAAS / IAAS Hybrid

Source Data SSISETL

Azure SQL DB / SQL DWData Warehouse

SSASCube

[Platform As A Service]

Azure Virtual Machine

Azure SQL DBStaging Database

On-Premise

VPN GatewayOr ExpressRoute

Azure

PDW / APS / SQL DWPDW / APS = Parallel Data Warehouse / Analytics Platform System

appliance, on premiseSQL DW = Same concept but in Azure. Each node is an Azure SQL DB

Azure SQL Data Warehouse

PAAS / IAAS Hybrid

Source Data SSISETL

Azure SQL DB / SQL DWData Warehouse

SSASCube

[Platform As A Service]

Azure Virtual Machine

Azure SQL DBStaging Database

On-Premise

VPN GatewayOr ExpressRoute

Azure

PAAS Solution

Source Data

AzureData

Factory

ELT

Azure SQL DB / SQL DWData Warehouse

PowerBI

[Platform As A Service]

Azure SQL DBStaging Database

On-Premise Azure

Cheaper Staging – Blob/Table

Source Data

AzureData

Factory

ELT

Azure SQL DB / SQL DWData Warehouse

PowerBI

Azure Blob StorageStaging Database

On-Premise Azure

Blob / Table Storage

Blob• Azure File System• Any file, any format• Unstructured• Cheap

Table Storage• Simple tables• No real indexing• No foreign keys• Basic data types• Basic querying• Structured• Cheap

Integrate Data Lake

Source Data

AzureData

Factory

ETLT

Azure SQL DB / SQL DWData Warehouse

PowerBI

Data LakeStaging

On-Premise Azure

Data Lake

U-SQL (T-SQL + C#)

Processing Engine

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Data Storage (HDFS)Structured Semi-Structured Unstructured

Dat

a La

ke

Anal

ytic

sD

ata

Lake

St

ore

Integrate Data Lake

SQL DWData Warehouse PowerBI

Data Lake Store

Staging

Data Lake AnalyticsTransform

Data Lake Analytics

Power User / Data Scientist‘big data’ analytics

Dashboards ExcelSelf-ServiceSQL Queries

/ Reports

HD Insight

Data Lake Analytics Vs HDInsightData Lake• U-SQL• (T-SQL + C#)

HDInsight• Hadoop• Map-Reduce

• Hive/Pig?

Integrate Data Lake

SQL DWData Warehouse PowerBI

Data Lake Store

Staging

Data Lake AnalyticsTransform

Data Lake Analytics

Power User / Data Scientist‘big data’ analytics

Dashboards ExcelSelf-ServiceSQL Queries

/ Reports

HD Insight

Real Time BI

SQL DWData Warehouse PowerBIData Lake

Staging & Transform

Data Lake AnalyticsOr HD Insight Power User / Data Scientist

‘big data’ analytics

Real-timeDashboards

ExcelSelf-ServiceSQL Queries

/ Reports

IoT or Event HubReal-time data

Real time

Batch Data

Stream AnalyticsReal-time insight

Machine Learning

SQL DWData Warehouse PowerBIData Lake

Staging & Transform

Data Lake AnalyticsOr HD Insight

IoT or Event HubReal-time data

Real time

Batch Data

Stream AnalyticsReal-time insight

Machine Learning

Azure Machine Learning

Cortana Analytics Suite

Cortana Intelligence

Cortana Intelligence - Batch

Cortana Intelligence - RealTime

Cortana Intelligence – Analytics

U-SQL / Data Lake Demo

U-SQL

U-SQL – User Defined Functions

U-SQL / Data Lake

U-SQL / Data Lake

Data Lake

U-SQL (T-SQL + C#)

Processing Engine

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Nod

e

Data Storage (HDFS)Structured Semi-Structured Unstructured

Dat

a La

ke

Anal

ytic

sD

ata

Lake

St

ore

In Summary…

• Real-time or Batch?

• Virtual Machine with On-Prem solution?

• Or genuine Azure solution?

• U-SQL & Data Lake <- Everything to Everyone!

• Missing:

• SSIS Data flow equivalent

• Analysis Services as a Service

Business Intelligence in Azure

[email protected] www.PurpleFrogSystems.com

@PurpleFrogSys www.PurpleFrogSystems.com/blog