BI ALM & TFS - Data United€¦ · BI ALM & TFS Microsoft Business ... •No need to buy pipelines in TFS 2015 ... •TFS Release management •More sophisticated •Better support

BI ALM & TFS

Microsoft Business Intelligence Application Life-Cycle Management

with Team Foundation Server.

(Good job there are acronyms)

About me: Dmytro Andriychenko

• Business Intelligence Architect, Developer, Data Scientist, etc.

• SQL Server BI Certified Expert

• Over 10 years data of data integration and analysis experience

• Lately focused on BI architecture, design and delivery

• Dmytro, Dmitriy, Dimitriy or just D – perfectly fine, I do not bite.

• Data United Ltd – all things data• Integrity

• Performance

• Insight

Why ALM: What can possibly go wrong?

• Knight’s tale: losing $172,222 per minute after a bad deployment

• Gartner: 40% of critical system outages are due to change management or deployment

• Career-limiting events:

• … year-end finance report figures were wrong?

What is successful ALM?

• Continuous, transparent, reliable integration and deployment

• Dev-Ops cooperation and collaboration

• Productive Agile Development

• Effective, sustainable management of• Work load/items

• Teams

• Product pipeline

• Infrastructure

Application Life-Cycle Management (ALM)

• Operate• Plan

• Deploy• Deliver

BI &DB Projects,

MS Test,Version Control

MsBuildTest Manager,

Release Manager

Bug Tracking

Backlog planning,

Sprint Board,BurnDown

chart

Sprint MonitorDev Ops

ALM I: What Planning? We are all Agile, right?

ALM I: Planning: the most important step

• Prior Proper Planning Prevents Poor Performance (Brian Tracy:10/90)

• Organise work into Features, PBIs(User Stories) and Tasks:

• Feature – project level item • Product Backlog Item (PBI): scrum – level container of work tasks

• Task: individual work item, user story, 2 days effort max

• Assign tasks to developers at the beginning of sprint & set duration

• Monitor and update tasks via task board & burn-down chart

• Facilitate communication by using the task board on agile ceremonies

Task Board

ProductBacklog

Item (PBI)or

User Story

Task

Burndown chart: progress against sprint plan

ALM 2: Delivery (Development)

• Version Control: TFS VC and/or git repository

• Database projects and dacpac deployments

• Database Unit Test Framework: MS Test

• Link Tasks to Check-Ins/Commits:

Keep them right

• Set compulsory Check-In comments

• Require Check-In to be allocated to a task – visibility

• Consider Gated Check-Ins if the builds are quick enough

ALM 2: Delivery (Be Prepared!)

• Configure Deployment Pipeline as early as possible• Helps envisage problems early

• Helps in planning and development

• Involve Ops early to avoid disappointment

• Deploy to Test/CI environment as often as possible• ideally every check-in/commit

• at least daily as practical

ALM 3: Deploy

• Use Deployment system: Release Manager or Octopus Deploy

• Automate Deployments: harness the PowerShell!

• Automate Approval Worklow: get your system do the legwork

ALM 4: Operate – Bug tracking

• Use TFS for logging Bugs and system issues

• Full traceability issue to resolution

• Web Interface

Let’s talk deployment!

• Release Manager• Better integration with TFS

• Work Items Tracking!

• Works well with BI projects• One-liner Build of any BI project with devenv.com

• Octopus Deploy • Free for 5 users/5 projects/20 tentacles

• Excellent RedGate integration

• Many other tools : XLDeploy, Cruise Control.NET, CHEF, etc

?

TFS 2017 Deployment

Drop location DeploymentAgent

TFS VC / Build / VS

Developer

Tester

E n v i r o n m e n t s

CodeBuild/Release Definitions

p i p e l i n e

Test Manager

OperationalSupport

Octopus Deploy architecture

Deployment ServerBuild/CI Server+VS + SSDT

Projects

Source Control

NuGetPackage

E n v i r o n m e n t s

Developer Tester Ops/Release Man

tinyurl.com/octo-build

T e n t a c l e s

L i f e c y c l e

Number of Agents

• TFS:• No longer a license issue: have as many as you need

• Secure TLS 1.2 communication with TFS Application Server

• Build agent should be separate from TFS

• Deployment agent must see the targets (firewall)

• Octopus:• 20 included in the free edition

Don’t like agents? Go agentless!

• Release Manager (DSC) or XL Deploy

• Utilize Windows Remote Management

• Can be perceived as more risky

• Harder to configure correctly

• More fun (for die-hard coders)

Concurrent Pipelines: you what?

• It is a number of Concurrent Releases actually running (not waiting)

• Every TFS 2017 server comes with one

• MSFT Rule of thumb: 1 for 10 devs ☺

• Extra pipeline for every Enterprise MSDN subscription

• No need to buy pipelines in TFS 2015 ☺

• Loosely equivalent to a project in Octopus

Deployment Tools summary

• Use what is accepted/supported in the company – don’t re-invent

• Octopus Deploy• Simpler to install and get started

• Can be cheaper if nothing is currently in place (free for 5x5x20 teams)

• Not MS BI – friendly: workarounds required

• TFS Release management• More sophisticated

• Better support for BI deployments

• Easy integration with Work Items

• Use if you already have it!

Deployment pipeline: logical steps

Developer Checks In Code

• Gated check-in for protection

• Check In Triggers for CI

Build is

Queued

• Build template

Auto-Deploy

to CI/Test

• Release template (TFS)

• Project + Lifecycle (Octopus)

Approve and release to Prod

• Deployment workflows (TFS)

• Dashboard (Octopus)

Build Release

Build: Off with his head?

• Headless build• Custom coding to build BI project

• Additional support overhead

• Devenv.com• Same as by developer (Visual Studio)

• Used in Azure

• Beware: Use 16.3 version of SSDT-BI (Build 14.0.60812) or older

• “Prime” SSDT-BI to make sure it knows what to do with SSAS Tabular

SSIS Tabular Visual Studio property

Deployment: each to their own

• Database projects: .DacPac (PowerShell or SQLPackage.exe)

• Analysis services: .ASDatabase (PowerShell or AS Deployment Wizard)

• Integration Services: .ISPac (SQL API or ISDeploymentWizard.exe)

• MDS: Import/Export (MDS API or MDSModelDeploy.exe)

• Reporting Services: .rds, .rdl (SSRS API or RS.exe)

Deploying Databases: script v dacpac

• Script: changes are manually scripted by a developer/tool• Visibility of what is going to change

• Finer control over the process (it seems)

• Works better for simple changes/schemas

• Dacpac: database model deployment• Consistency: fully transactional

• Designed to protect data integrity

• Repeatable

• Free of side-effects

Deploying Databases: SqlPackage.exe

• Hidden Gem of SQL Server Data Tools

• Program Files (x86)\Microsoft SQL Server\120\DAC\bin

• Can use the same profile file as in Visual Studio with your dacpac

• Very easy to use:

sqlpackage.exe /action:publish

/sourcefile <dacpacFilename>

/TargetServerName <serverName>

/TargetDatabaseName <DBName>

/Profile <profileFilename>

Deploying Databases: PowerShell

• # Add the DLL

• Add-Type -path "C:\Program Files (x86)\Microsoft SQL

Server\120\DAC\bin\Microsoft.SqlServer.Dac.dll"

• # Create the connection string

• $d = New-Object Microsoft.SqlServer.Dac.DacServices

"server=(local)"

• # Deploy the dacpac

• $d.Deploy(“C:\Path\AW.dacpac”, “AWDW”, $true, $options)

• See full working script at dataunited.co.uk/octo-dacpac

Deploying Cubes: Analysis Services Database

• Microsoft.AnalysisServices.Deployment.exe "C:\...\AdventureWorksCube.asdatabase" /s

• The path must be absolute!

• Only full deployments (all changes) are supported this way. If partial deployment is required, use API, or BISM Normaliser for Tabular.

• Use DeploymentWizard GUI to configure “Input Files”:

Deploying Cubes: Analysis Services XMLA

• Invoke-ASCmd PowerShell cmdletInvoke-ASCmd

–InputFile:”C:\MyFolder\DiscoverConnections.xmla”

• All paths must be absolute!

• If you get “The term 'invoke-sqlcmd' is not recognized …”:• Install PowerShellExtensions for the version of SQL Server on targets

• Run Import-Module SQLASCMDLETS to make use of them (comes with SQL Server Client Tools)

Deploying ETL: SSIS

• Deployment artefact: .ispac file

• ISDeploymentWizard.exe

/Silent /SourceType:File /SourcePath:“C:\...\AdventureWorks.ispac" /DestinationServer:“localhost" /DestinationPath:"__DestinationPath__“

• Consider a separate database project for server-level components• Environments (remember to scripts when you create them)

• Agent Jobs

• Security, etc

Other deployment

• SSRS• Check out “Continuous Integration for Reporting Services”

by Nat Sundar on the last SQL Bits: http://tinyurl.com/SSRS-CI• Includes deploying, refreshing and testing SSRS reports with nUnit

• MDS: MDSModelDeploy.exe• Run as administrator and on the MDS server (another Agent)

• Careful with model-only deployments – always check the results!

• Everything else: use the Power in PowerShell!• Just let it use the latest .Net framework: search for OnlyUseLatestCLR

Deployment summary: PowerShell or CMD?

• Complex projects require PowerShell: to many moving parts

• Simpler projects are easier to do with CMD

• If you can use CMD – do it!

• Both require SQL Server Client Tools installed on the Deployment Agent

Demo of CI using TFS

• I wish I can cover both, but there is no time…