Business Analytics: Collapsing The Stack Timo Elliott September 2011
Jul 09, 2015
Business Analytics:
Collapsing The Stack
Timo Elliott September 2011
2
Business Analytics Has Struggled to Keep Up
“Where are you going? Ah -- If I were you, I wouldn’t start from here”
3
Reporting
“Typical” Business Intelligence Today
Slow
Painful
Expensive
Operational Data Store
Data Warehouse
Indexes
Aggregates
DataBusiness Applications
Copy
ETLCalculation EngineBusiness Intelligence
Query Results
Query
Slow
Painful
Expensive
Operational Data Store
Data Warehouse
Indexes
Aggregates
DataBusiness Applications
Copy
ETL
Calculation EngineBusiness Intelligence
Query Results
Query
Data
Marts
4
It’s like an Onion…
the more layers there
are, the more it makes
you cry
5
What’s the Problem?
Slow Disks & CPUs
I/O Bottleneck
Expensive Memory
Optimized for Transactions
BI is an Afterthought
30 Year-Old Database Design Principles
6
A Revolution…
Credit Suisse, “The Need for Speed”
7
Today’s Disks Can’t Keep Up With Processing Power
8
In-Memory Computing Costs Have Plummeted
BT Tower
152m
Cost of 1 Mb of
memory in 2000: ≈£1
9
In-Memory Computing Costs have Plummeted
Cost of 1 Mb of memory
today: ≈ ½ p
My daughter:
1.30m
And shrinking, and
shrinking, and shrinking….
Price/performance of
in-memory has
DOUBLED in last 9
months
10
In-Memory Computing
Operational Data Store
Data Warehouse
Indexes
Aggregates
DataBusiness Applications
Copy
ETL
Calculation EngineBusiness Intelligence
Query Results
Query
Up to 1,000x faster
No optimizations requiredData
Marts
11
Row vs. Column Databases
My Filing System
My Wife’s Filing System
Row-based Column-based
12
Row-Based Data
Wasted space,
and a full scan to
aggregate any
particular field
13
Column Data
More efficient data storage, better compression, faster queries
14
Data WarehouseData Warehouse
Column Databases
Operational Data Store
Data Warehouse
DataBusiness Applications
Copy
ETL
Calculation EngineBusiness Intelligence
Query Results
Query
Up to 1,000x faster
More data in less space
15
Data Warehouse
Massively Parallel Hardware
Operational Data Store
DataBusiness Applications
Copy
ETL
Business IntelligenceQuery Results
Query
Up to 1,000x faster
Optimized for hardware – especially good for column stores
Calculation Engine
16
In-Database Processing
user changes
a plan value
52 weeks x 500 branches = 26000 values
26000 database writes 1 database write
17
A Database Designed for Business
Volume Driver
Cycles
Driver
Forecast Driver
Forecast Agents
Grow
Seasonal Complex
Assortment Planning
Cumulate
Days
Days Outstanding
Discounted Cash Flow
De-cumulate
Delay
Delay Debt
Delay Stock
Annual Depreciation
Annual Depreciation
Diminishing Balance
Depreciation
Sum of Year Depreciation
Year To Date Statistical
YOY/ YOY Difference
Forecast Dual Driver
Forecast Sensitivity
Feed
Feed Overflow
Forecast
Funds
Future Value
Inflated Cash Flow
Internal Rate of Return
Moving Median
Number of Periods
Net Present Value
Outlook
Payment
Present Value
Lag
Last
Lease
Lease Variable
Linear Average
Forecast Mix
Moving Average/Sum
Proportion
Rate
Repeat
Seasonal Simple
Seasonal Simulation
Stock Flow
Stock Flow Reverse
Stock Flow Batch
Time
Time Sum
Max Value
Minimum Value
Transform
Rounding
Up until now, there’s been a false separation between application logic and
database functionality
18
In-Database Analytics
Forecasting ClusteringAnomalies
Influencers Trends Meaningful or Random?
19
Data Warehouse
In-Database Analytics
Operational Data Store
DataBusiness Applications
Copy
ETL
Business IntelligenceQuery Results
Query
Up to 1,000x faster
Push processing down to dedicated hardware, less traffic
Analytic Appliance
Calculation Engine
20
Integrating Flows of Data
Incremental loads, replication
21
Integrating Flows of Data
22
Streaming Data
23
Real-Time Data
Operational Data Store
Copy
ETL
Real-time replication — why have a separate operational data store?
DataBusiness Applications
Analytic ApplianceBusiness Intelligence
25
The Basis For Applications of The Future
Copy
Business Applications
Analytic ApplianceBusiness Intelligence
Use a single appliance for both analytics and applications
Data
26
Applications of the Future
27
Virtuous Circle of Technology
In-Memory
Columnar Databases
Hardware Acceleration
Calculation Engine
Columnar storage
increases the
amount of data that
can be stored in
limited memory
(compared to disk)
Column databases
enable easier
parallelization of
queries
In-memory processing
gives more time for
relatively slow updates
to column data
In-memory allows
sophisticated calculations
in real-time
Hardware acceleration
makes sophisticated
calculations like
allocations possible
Each technology works well
on its own, but combining
them all is the real
opportunity — provides all of
the upside benefits while
mitigating the downsides
28
Extended Architecture
Business Applications
Analytic ApplianceBusiness Intelligence
Cloud computing
Unstructured and personal data
Mobile revolution
Collaboration
29
In-Memory Computing is Like Digital Photography
A transformative
technology that slowly
but surely upturns the
whole industry
Faster, Easier, More
Convenient
Evolved Faster Than
The Alternatives
30
It’s All About Flexibility and Evolution
“It's not the strongest that
survive, nor the most
intelligent, but the ones
most responsive to
change.”
Charles Darwin
31
Reality Is, and Always Will be, Messy
Different information
sources
Different
levels of
expertise
Different access
devices
Different time
horizons
Different levels of
analytic need
Different
project phases
Risk
Politics
But new architectures mean simplification and new opportunities
32
What About Flash Disk / SSDs?
15X
9000X
16X
Cost-effective, but not a revolution
33
What About Big Data / NoSQL / Hadoop?
Thanks!
Email:
BI Blog:
timoelliott.com
You Should Follow Me on Twitter: @timoelliott