Tdwi agile data warehouse - dv, what is the buzz about

R.D.Damhof

Data Vault, What is the buzz about

TDWI München June 18, 2012 Ronald Damhof

Agile Data Warehousing

R.D.Damhof

“Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software” Agile Manifesto, 2001

Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern,

Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

R.D.Damhof

Source

Source

‘Semantic gap’

‘Calculating risk’

‘Yield modules’

‘Customer ��segmentation’

R.D.Damhof

Everybody mines their own data Everybody enriches their own data

Everybody uses their own data User = Developer

With his selfmade tools Data quality determined by the individual It’s a grind – limited reusability Leadtimes unpredictable No management

R.D.Damhof

Lets ‘order’ an information product And hire a master/expert Separation between user/developer Developer/expert mines the data The information product = custom made

Data quality is mostly dependable on the developer/expert

Leadtimes unpredictable

Still not much reusability

R.D.Damhof

A central department who knows what information you need

That assembles information products, ready to be used for you

‘I now what you want’ – black

Efficiency is the name of the game

At least I got something, but it does not comply - even remotely - to my needs

Even worse; the guild-days are still there – the expert is now submerged, but needed to get the data you actually need. Introduction of management – you want something? Please apply in 3-fold…

R.D.Damhof Stephen Denning (2011) – Radical Management

Creating information products, the moment they are asked for Against quality criteria which are in line with the expectation of the customer Empower the customer with skills and facilities to be more self sufficient Minimize ‘data’-stock as much as possible Embrace new wishes and changes required by the customer The customer is the most important part of the production process

R.D.Damhof

A modern data management environment: The ‘Supermarket’

The ‘Restaurant’

The ‘Do it yourself buffet’

R.D.Damhof

R.D.Damhof

Push characteristics §  Mass production §  Known specifications, operational definitions, standards §  Repeatable, predictable, & even better; uniform process §  Part of the system that needs statistical control §  Inventory allowed/necessary §  Supply driven §  Reliability over flexibility

Pull characteristics §  Just in time § Demand driven § Build to order § Preferably no inventory § Flexibility over Reliability

R.D.Damhof

Back to the issue at hand……

§  What: the ‘production process of data’ §  Where: Coordination - Local versus central §  How: System Engineering - Systematic vs. Opportunistic §  What principles guide us - leading principles

R.D.Damhof

1

2

1

2

3

4

1

2

3

4

1

2

3

4

1

2

3

4

Data & fu

nc.o

n service

3

4

End-‐user (Local)

Recipient

Inform

a.on

Delivery process

Generic proces (Central)

Data sources (internal & external)

4. Generate Informa.on products

3. Enrich and cleanse data

2. Register & Standardize

1. Get the raw uncut data

Informa.on Delivery Proces

Local vs Central deployment

R.D.Damhof

IT Development

Delegated Development

Selfservice Development

Development line discipline (OTAP) Developers at a distance from users

Mutually dependent/ within frameworks Heavy separation of function

Lightweight development process Minimum of specialisation/ distinction of roles

Self-sufficient/ limited freedom

Manoeuvrability (opportunistic approach)

Sustainability (Systematic approach)

Ad-hoc development proces Developer=user

Self-sufficient/ great degree of freedom Very broad tasks

System Engineering - Systematic vs. Opportunistic

R.D.Damhof

Adaptible

Sustainable

Decoupled

Centralized

Compliant

Standardized &

Industrialized Effective

Leading principles

R.D.Damhof 15

Enterprise Data Warehouse

BI Apps Analysis

BI apps Reports

BI Apps Ad-‐hoc

Company xxx data management Domain

Data, ‘What’ Func.on, ‘How’ ‘Where’, ‘Whom’

Busin

ess V

iew

Sources Source store

1 2 3 4

R.D.Damhof 16

Sustainable

Compliant

Decoupled

Standardized

Centralized

Adaptable

Effec.ve

Source to product

Sourcestore to product

Sourcestore to BV EDW (DV)

R.D.Damhof 17

Enterprise Data Warehouse

BI Apps Analysis

BI apps Reports

BI Apps Ad-‐hoc

Company xxx data warehouse & Business Intelligence Domain

Data, ‘What’ Func.on, ‘How’ ‘Where’, ‘Whom’

Busin

ess V

iew,

Data fe

eds Sources Source store

1 2 3 4

R.D.Damhof 18

Administra.ve process

Data & Informa.on recipients

Informa.on Delivery Process

AXain

Register & Standardize

Enrich

Generate& Distribute

Proces

Decision-‐ & control

Systems (internal & external)

DV based Data

Warehouse

Informa.on products

Data products

Compliance repor.ng

Supply chain op.miza.on

Staging

Risk Management

Performance Management

Fraud detec.on

Market basket analysis

Business rules

PDCA

Control / Metadata

Pull

Push

Push

Why DV?

R.D.Damhof 19

Metamodel driven automa.on -‐ Models (process, rules and data) determine the metadata, the metadata determines the automa.on ar.facts -‐ Aim is to be 100% declara.ve -‐ It can not be generated all, specific tailored metadata will remain necessary

Metadata driven automa.on -‐ Inputs: Source model(s), target model, Template Design, Naming conven.ons -‐ Advanced inputs: Normaliza.on preferences, Ontologies Taken from Dan Linstedt’s blog post: hXp://danlinstedt.com/datavaultcat/code-‐genera.on-‐for-‐data-‐vault-‐not-‐as-‐easy-‐as-‐you-‐think/

Template driven automa.on -‐ In the most basic forms; documenta.on -‐ describing a paXern -‐ More advanced; genera.ng XML code for 2nd gen. ETL tooling -‐ Vb -‐ hXp://www.grundsatzlich-‐it.nl/bi-‐tools-‐templator.html

Data Vault implementa.ons

R.D.Damhof 20

My PoV about (Data Vault) automation Tooling

§ Generation is an aid, not a goal in itself Do not accommodate the principles to fit the tool.... Look for decoupling

§ Truly understand the mechanics - handcraft it first! Invest in proper education and learning Invest in ‘getting ready’ time Involve your ‘customers’ from the start

§ PoC, PoC, PoC

§ Deliver, Deliver, Deliver

R.D.Damhof

Agility & Data Vault (1)

Why is it that you can build and deploy extremely small particles in Data Vault and not in other approaches, without having an increase in the overhead and coordination of these particles? In other words; 'Divide and Conquer to beat the Size / Complexity Dynamic’

R.D.Damhof

Why is it that you can re-engineer your existing model and guarantee that the changes remain local? Something that is hugely beneficial in data warehouses that - by definition - grow over time.


R.D.Damhof

Why is it that - as your (Data Vault based) data warehouse grows - your costs grow ‘merely’ in linear fashion initially, and as you approach the end state marginal growth in cost decreases exponentially.


R.D.Damhof

Data Vault as-such is not Agile, it is the development process that needs to be agile, DV merely supports

the agile development process.

24

“Our highest priority is to satisfy the customer through early and continuous

delivery of valuable software”

Agile Manifesto, 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham,

Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas

R.D.Damhof

Data Model Time Line Historic Overview

© (Linstedt, Graziano, & Hultgren, The New Business Supermodel, The Business of Data Vault Modeling, 2008, p. 36)

§  Created By Dan Linstedt §  Released in 2000 §  Formally Introduced in the Netherlands in 2007

§  First DV Book: The Business of Data Vault Modeling 2008 §  First (Dutch) User group in 2010 §  Technical book from Dan Linstedt in 2011

R.D.Damhof

Application��Architecture

R.D.Damhof

Top Down Approach

R.D.Damhof

Bottom Up Approach

R.D.Damhof

Bottom Up Approach

R.D.Damhof

Bottom Up Approach

R.D.Damhof

Bottom Up Approach

R.D.Damhof

Irony

R.D.Damhof

Hybrid Approach (Data Vault)

R.D.Damhof

R.D.Damhof

ETL/Load Architecture -  100% of the data (within

scope) 100% of the time -  Source driven /Auditable: -  “Fact Oriented” -  Template/metadata driven -  No Business Rules

Kimball or Inmon ETL -  Complex ETL -  Truth oriented -  Business Rules before EDW

Pictures: Genesee Academy ©

R.D.Damhof 36

Data Vault

Business Transac.on System


Structure transforma.on Hub = business keys

Datasets

Business rule execu.on Structure and value transforma.on

Staging Out

Classic Data Vault Application Architecture

Adaptable Sustainable Compliant Decoupled Effec.veness Standardized Centralized

Rule Vault

Generic Business Rules

? ?

R.D.Damhof

Data Vault Application Architecture

§  Central EDW §  Business rules downstream §  Incremental/Non destructive Loading §  100% of the data (within scope) 100% of the time §  Auditable/Partly source driven

R.D.Damhof

Modeling

R.D.Damhof

R.D.Damhof

R.D.Damhof

R.D.Damhof Pictures: Genesee Academy ©




R.D.Damhof

Data Vault Constructs


R.D.Damhof



R.D.Damhof



R.D.Damhof

Core Components

R.D.Damhof

Data Vault Core Components


R.D.Damhof

Data Vault Core Components


R.D.Damhof

Hubs


R.D.Damhof

Hubs


R.D.Damhof

Hubs


R.D.Damhof

Satellites


R.D.Damhof

Satellites


R.D.Damhof

Links


R.D.Damhof

Links


R.D.Damhof

Loading

R.D.Damhof

HUB load


R.D.Damhof

INSERT INTO customer_hub (cust#,load_dts,record_src) SELECT source.customer#, @load_dts, @record_src FROM source_customer AS source WHERE

NOT EXISTS (SELECT * FROM customer_hub AS hub WHERE hub.customer#=source.customer#)

HUB load


R.D.Damhof

Loading a Link

Link Load


R.D.Damhof

Link Load

INSERT INTO custcontact_link(cust_id,contact_id,load_dts, record_src) SELECT source.customer#, @load_dts, @record_src

FROM source_table AS source INNER JOIN contact_hub AS contact ON

contact. contact#= source.contact# INNER JOIN customer_hub AS cust ON

cust. customer#= source.customer# WHERE NOT EXISTS (SELECT * FROM custcontact_link AS link WHERE link. contact_id= contact.id and link.cust_id= cust.id)


R.D.Damhof

Loading a Satellite

Satellite Load


R.D.Damhof

Satellite Load

INSERT INTO customer_sat (hub_id,load_dts, name,record_src) SELECT hub.id, @load_dts, source.cust_name, ,@record_src

FROM source_customer AS source INNER JOIN customer_hub AS hub ON

cust.customer#= source.customer# # INNER JOIN customer_sat AS sat ON sat.id= hub.id# AND sat “Is most recent” AND

sat.name <> source.name


R.D.Damhof

Data Vault Loading Paradigm


R.D.Damhof

Top 10 Rules for Data Vault Modeling


R.D.Damhof 68

Why is it that you can build and deploy extremely small particles in Data Vault and not in other approaches, without having an increase in the overhead and coordination of these particles? In other words; 'Divide and Conquer to beat the Size / Complexity Dynamic’

Why is it that you can re-engineer your existing model and guarantee that the changes remain local? Something that is hugely beneficial in data warehouses that - by definition - grow over time.

Why is it that - as your (Data Vault based) data warehouse grows - your costs grow ‘merely’ in linear fashion initially, and as you approach the end state marginal growth in cost decreases exponentially.

Agility & Data Vault - recap (1)

R.D.Damhof

➡  Mass production

➡  Known specifications, operational definitions, standards

➡  Repeatable, predictable, & even better; uniform process

➡  Part of the system that needs statistical control

➡  Inventory allowed/necessary

➡  Mainly supply driven

➡  Reliability over flexibility

Remember the Push characteristics Data Vault

Data Vault

Data Vault

Data Vault

Data Vault

Data Vault

Data Vault

Automation of a Data Vault ‘production process’ is just common sense

Agility & Data Vault - recap (2)

R.D.Damhof

Bonus Slides��Forks and mutations in DV ‘evolution’

R.D.Damhof 71

Data Vault



Structure transforma.on Hub = business keys

Datasets

Business rule execu.on Structure and value transforma.on

Staging Out

Type 1 - Classic Data Vault

Adaptable Sustainable Compliant Decoupled Effec.veness Standardized Centralized

Rule Vault

Generic Business Rules

? ?

R.D.Damhof 72

Staging Vault



Structure transforma.on No integra.on, Hub=surrogate keys Persis.ng staging in DV format

Business Data Vault

Business rule execu.on Integra.on DV modelled

Staging Vault

Data Marts

Structure transforma.on

Type 2 - Source Data Vault

Sustainable Compliant Decoupled Standardized Centralized

? ? ? Adaptable Effec.veness

R.D.Damhof 73

Source

Source

100% Seman.c gap

Source

Source

100% Seman.c gap

Staging DV

Staging DV

Business DV

Integra.on, cleansing, consolida.on Business rule execu.on upstream ?? DV modelled

S.ll the source

R.D.Damhof 74

Source

Source

100% Seman.c gap

Source

Source

100% Seman.c gap

Staging DV

Staging DV

Business DV

Integra.on, cleansing, consolida.on Business rule execu.on upstream ?? DV modelled

S.ll the source

Source

Source Data Warehouse

R.D.Damhof

Wanna know more? § Training & certification: www.geneseeacademy.com

§ Books: ‘Super Charge Your Data Warehouse: Invaluable Data Modeling Rules to Implement Your Data Vault’ – D.Linstedt / K.Graziano

§  Linkedin: Data Vault Discussions (approx. 800 members)

§ Niche non-commercial conferences; www.dwhautomation.com

§ Many blogs, articles, presentations on the World Wide Web

§ The best way to learn; try it, make some code, experience, engage

R.D.Damhof 76

Drs. Ronald D. Damhof

Blog hXp://prudenza.typepad.com/ hXp://www.b-‐eye-‐network.com/blogs/damhof/

Linkedin hXp://nl.linkedin.com/in/ronalddamhof

Email [email protected]

TwiXer RonaldDamhof

Skype Ronald.Damhof

Mobile +31(0)6 269 67 184

Others Informa.on Quality Cer.fied Professional (IQCP) Data Vault Cer.fied Grand Master Cer.fied Scrum Master Member of the Boulder BI Brain Trust (#BBBT)

Ronald Damhof is an independent prac..oner in the field of data management and decision support. Graduated in 1995 in the study of Economics. Since 1995 he worked as a prac..oner into the field of Informa.on Management with a focus on decision support and data management, trying hard to enhance the rigor and relevance in these fields by combining scien.fic research with the everyday challenges of the prac..oner. Ronald is mainly hired by customers in the role of business/IT architect, auditor, coach & trainer. He blogs on B-‐Eye-‐Network.com as well as his own blog, is a member of the pres.gious BBBT, wrote several ar.cles regarding decision support architectures and is a researcher in the field of Informa.on Management. Although Ronald likes to work with theore.cal grounded research and proven prac.ces, Ronald is not a 'white paper' architect; put your money where your mouth is, is his moXo. He likes to see architectures 'live' in enterprises, not just write about it. In most organiza.ons his role extends architecture onen. In truely agile spirit the roles he plays depend on the context of the client; he can be a missionary (selling the value), a project manager (geong it done), a scrum master (removing impediments), specialist (educa.ng hardware peeps, data architects, data logis.cs etc.) or a leader.

Thank You

Tdwi agile data warehouse - dv, what is the buzz about

Technology

data informa

cleanse data

process informa

push data products fraud

reliability r

leading principles r

models process

information products