Deep Dive Data Management Gateway - SQLSaturday Edinburgh

Data Management Gateway

Deep Dive

Thanks to our Main Sponsors:

Say Thank you to Volunteers:

They spend their FREE time to give you this event.

Because they are crazy. Because they want YOU

to learn from the BEST IN THE WORLD.

Jean-Pierre RiehlPractice Manager Data & BI

@djeepy1 - http://blog.djeepy1.net

Who am I ?

Pure-Player Microsoft Self-Service BI experts « Agile BI » projects Data Strategy Hybrid-BI

MVP SQL Server

French PASS Chapter Leader

La communauté Data Microsoft

Webcasts, Conférences, Afterworks

.Pro

Save the DateSeptember, 5thCampus SUPINFOTour Montparnasse, Paris

Self-promotion

An hybrid story…

Microsoft Cloud-First strategy…

But your data remains On-Prem…

Here it comes DMG

DMG-aware tools

Power BI for O365

PowerBI.com

Azure Data Factory

Power BI

ExcelPower Query

Power Pivot

Power View Power Map

Power BI Tenant SQL

Cloud On-Prem

Oracle

…Power BI Admin Center

Data Catalog

Q&A

Power BI Sites

Data Refresh

Indexing / Search

OData Access

PowerBI.com

Live Power BI reports &

dashboards

LiveQuery

Scheduled Refresh

Cloud

On-premises

Azure Data Factory

Hybrid features

Data Refresh Data Movement Live Query Indexing Sources OData Access (Data Virtualization)

One DMG to rule them all

DMZ

SQL

Cloud On-Prem

Oracle …

Power BI for O365

DataManagement

Gateway

OnPrem Data Sources

PowerBI.com

Azure Data Factory

Excel

DMG DETAILS

Versions of DMG

There are multiple versions of DMG

Data Management Gateway Power BIADF 1.4.5549.1 24/3/2015

Analysis Services Connector Power BI v2 1.4.5576.2 17/4/2015

Breaking News: 1.4.5612.1 (released 26/5)

What’s that

A Windows Service DIAHostService

A configuration application

Managed by the cloud Power BI Admin Center Azure Portal (new version)

demoFIRST LOOK AT DMG

Data Sources SupportedSource Authentification

SQL Server / OracleSybase, Teradata, DB2, MySQL Windows, Basic

Files (CSV, XML, Text, Excel, Access) Folders Windows

SharePoint List (Online)SharePoint List (On-prem)

AnonymousAnonymous, Windows

OData Feed Anonymous, Windows, BasicKey (Data Market)

Azure Marketplace Key (Data Market)

Azure HDInsightAzure Blob StorageAzure Table Storage

Key (Azure Account)

SQL AzurePostgreSQL Basic

Teradata, Sybase, DB2, MySQL Windows, Basic

SSAS Tabular (Live Query) Windows

+SSIS Data Feed

Data Sources supported

Details on data sources supported https://support.office.com/en-us/article/Supported-Data-Sources-an

d-Data-Types

Prequisites Beware of versions Beware of types Drivers installed https://support.office.com/en-us/article/Data-source-prerequisites-P

ower-Query

HOW DOES IT WORK

Cloud On-Prem

Data Refresh explained

Power BIOnPrem Sources

DMG

Azure Service Bus

0Register DMGRegister Data Sources

1Ask for Data RefreshCheck permissionsFind Gateway

4 Execute Query

5 Get Data

6 Compress, Chunk

Queuerequest2

3 Readrequest

GatewayCloud Service

7 Update Model

Cloud On-Prem

OData access explained

Power BIOnPrem Sources

DMG

0 Index Sources

2Resolve Data SourceRedirect to local Gateway

4 Execute Query

3 Request Data

1 Search Source

Excel

Power Query

Hybrid Proxy

demoDMG IN ACTION

Data Source Discovery

Data Sources are declared on « Admin Center » They are associated to a Gateway

Gateway Cloud Service resolves Data Source and find right Gateway

Resolution = Provider + Server + Catalog Warning: Case sensitive

demoDATA SOURCE DISCOVERY

SECURITY

Excel

Power Query

Power BI

DMG

Data sources• SQL (1433)• FileShare• Etc.+DNS resolution

Azure Service Bus (outbound)• 9350-9354 / 5671,5672Power BI Admin Center (outbound)• 80/443

Direct HTTP (inbound)• HTTPS (8050)• HTTP (8051)

HTTPS (443)

Cloud

On-Prem

Network Security

Credential Manager

Identity Management

2 flavors : Impersonation by DMG Impersonation by Data Source

How is query authentified on Data Source ?

Impersonation by DMG

Connection identity is impersonated by DMG Peer-to-peer connection

Credentials are stored by DMG for each source Credentials can be stored encrypted locally Credentials can be stored encrypted on Cloud

Credentials are set up locally Credential Manager is a Click Once application Credential Manager discuss with DMG

demoCREDENTIAL MANAGER

Impersonation by Data Source

Used by Live Query Keep identity of original user Use the EffectiveUsername method

DMG authenticates to SSAS DMG passes EffectiveUsername property SSAS executes the query as if it was original

user DMG must be SSAS Admin

demoEffectiveUsername

HIGH AVAILABILITY &DISASTER RECOVERY

As easy as scale-out

DMGs

Cloud

On-Prem

Power BI

…Instance 1 Instance 2

Server A Server B

HA & DR

Gateways and Data sources are managed by Cloud

Credentials are encrypted by MDG Use of Certificate, self-generated or yours Need Certificate when registering a new instance Warning when stored localy

Load Balancing

demoHA & DR

Going further

Express Route Mashup Container (aka Power Query) Monitoring & Logging

Resources

Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help-5e391ecb-500c-47a3-bd0f-a6173b541044

Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx Supported Data sources :

https://support.office.com/en-us/article/Supported-Data-Sources-and-Data-Types DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931 How does DMG work - sequence diagram : https://

support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway-078D306D-2D28-409C-962B-D80B69A1D0B2

Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653 Store encrypted credentials in the cloud for business continuity : https://

support.office.com/en-us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886-da51b8cadc5b

Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint-Conference/2014/SPC397

Mini Nay & Luming Han Session at PASS Summit : http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231

@djeepy1 - http://blog.djeepy1.net

Any questions ?

Thank You!