7 Dimensions of Agile Analytics by Ken Collier

7 DIMENSIONS OF AGILE ANALYTICS Ken Collier, PhD Director, Agile Analytics @theagilist #thoughtworks

1

Busi

ness

Val

ue

Analytical Complexity

What happened?

Descriptive Analytics

Why did it happen?

Diagnostic Analytics

What will happen?

Predictive Analytics Can we influence

what happens?

Prescriptive Analytics

Busi

ness

Val

ue

Analytical Complexity

What happened?

Descriptive Analytics

Why did it happen?

Diagnostic Analytics

What will happen?

Predictive Analytics How can we

make it happen?

Prescriptive Analytics

3

Business Intelligence

Data Science

THE DIFFERENCE

Data Engineering

Lean Learning

Streaming data pipelines &

adaptive architectures

Continuously challenge your assumptions by

measuring results.

Discovery of patterns and

signals hidden in data

Agile Delivery

Data Science

Deliver business value early and

often. Build your platform over time,

not all up front. Your Business

Questions

=

Fast results & Early Value

Data Guided Market

Advantage

Agile Analytics

Data Engineering

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Advanced Analytics

Agile Analytics

Data Engineering Solutions

Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Advanced Analytics

Adaptive Architecture

Streaming Data

Polyglot Persistence

Strictly Private and Confidential © 2015 ThoughtWorks, Ltd. All rights reserved.

REMOVING THE DATA BOTTLE NECK

Data Warehouse Incoming data is cleaned and

organised into a single schema up front.

Data Lake Incoming data goes into the lake

in its raw form.

Long development times to create new value from data.

Analysis activities are distributed across technologists and business

users.

Analysis is done directly on the curated warehouse data.

Data is selected, structured, and organize as needed, when

needed.

DATA LAKE DONE RIGHT

8

Operational systems communicate directly

with each other via services

Operational systems push data to the lake

via topical queues

Data scientists explore the lake for

potential insights

Lakeshore marts and services curate and

organize the data for self-service analysis

Multi-tiered data lake for processing,

distribution, serving

ADAPTIVE ARCHITECTURE PRINCIPLES

9

Enable low latency data streaming

Store raw, low-level, historized data

Enable NoSQL presentation

Enable inexpensive scaling

Simplify data ingestion

Drive logic closer to the business

Enable emergent design

Enable easy recreation of data

DATA ENGINEERING CONCERNS

10

Streaming

Distributed MPP architecture

Data Strategy Elastic cloud computing

Reactive architecture

Master data

Data Governance

ETL techniques

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Discover & Explore

Analyze & Act

Data Convergence Analytical Divergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Analytical Opportunities

HOW DATA SCIENCE WORKS Can we anticipate what the customer

will want to do next?

THE “DATA SCIENTIST”

Machine Learning Statistical Modeling

Artificial Neural Networks

Decision Tree Learning

Support Vector Machines

Unsupervised Learning

…and many more…

Bayesian Classification

Monte Carlo Simulation

Logistic Regression

K-Nearest Neighbor

…and many more…

Feature Engineering

Feature Extraction

Dimension Reduction

Domain expertise

Programming Skills

Functional Programming

Data “Wrangling”

Map/Reduce, SQL, & NoSQL

Objective Truth

Discoverable Truth

Uninterpretable

Irrelevant Noise

Not Actionable

Actionable Signals

MAKING

“BIG DATA”

INTO

“LITTLE DATA”

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Data Engineering

Volume Velocity

Variety

Adaptive Architecture

Streaming Data

Polyglot Persistence

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Continuous Integration

Collaboration Evolve

Continuous Delivery

Hypothesis

Build Learn

Measure

Data Reduction

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

drones.pitchinteractive.com

Data Visualization

drones.pitchinteractive.com

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Visual Storytelling

Typical Timeline

3-6 months 1-2 months 2-4 months

22

Data Convergence Analytical Divergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Analytical Opportunities

CONVENTIONAL DATA SCIENCE If we knew X, we could do Y

Analytical Divergence

Analytical Opportunities If we knew X, we could do Y

Data Convergence

Discover

Harvest

Filter

Integrate Augment Analyze

Act

Repeat this cycle solving small problems every few days

LEARN

MEASURE

BUILD

LEAN DATA SCIENCE

Advanced Analytics

Agile Analytics

Solutions Thinking

Ethics

Agile Delivery

Impact

Reflect & Improve

Collaborate Evolve

Deliver

Data Engineering

Adaptive Architecture

Streaming Data

Polyglot Persistence

Data Science

Machine Learning

Statistics

Visual Storytelling

Lean Learning

Hypothesis

Build Learn

Measure

Retain high value customers

Problem solved or continue?

High value business goal

What’s the smallest, simplest thing we can do?

Is it useful & actionable? Repeat!

What leads to customers leaving?

LIKE THIS EXAMPLE… Common features of

defectors?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Retain high value customers

High value business goal

Like this example…

What’s the smallest, simplest thing we can do?

Retain high value customers

Like this example… Common features of

defectors?

Is it useful & actionable?

Retain high value customers

Like this example… Common features of

defectors?

Repeat! Retain high value customers

Like this example… Common features of

defectors?

Shopping behaviors of defectors?

Retain high value customers

Like this example… Common features of

defectors?

What leads to customers leaving?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Problem solved or continue?

What leads to customers leaving?

Like this example… Common features of

defectors?

Shopping behaviors of defectors?

What do defectors say about us?

Customers’ sentiment before defecting?

What encourages customers to stay?

Do incentives reduce defection rates?

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Data Engineering

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

QUESTIONS FIRST, DATA SECOND

34

“We built a platform for self service, and now we’re trying to get the business to use it.”

From this… …to this

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Privacy Controls Radical Transparency

Data Democracy

Open Data

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

38 http://bit.ly/DataAndPrivacy

Jonny Leroy, NA Head of Technology, ThoughtWorks

39

40

41

42

43

44

45

46

PRIVACY IS DEAD

DON’T BE EVIL

47

48

49

50

51

52

Advanced Analytics

Data Science

Visual Storytelling

Machine Learning

Statistics Agile Analytics

Solutions Thinking

Ethics

Agile Delivery Lean

Learning

Impact

Hypothesis

Build Learn

Measure

Insight

Knowledge

Action

Disruption

Business First

Evolve the Platform

Monitor & Measure

Privacy Controls Radical Transparency

Data Democracy

Open Data

Data Engineering

Volume Velocity

Variety

Adaptive Architecture Streaming

Data Polyglot

Persistence

Reflect & Improve

Collaborate Evolve

Deliver

Ken Collier, Director, Agile Analytics kcollier@thoughtworks.com

Value Creation

Cool New Technologies +

Sophisticated Analytics +

Lean Learning Principals +

Fast Agile Delivery =