1 © 2014 SAP SE or an SAP affiliate company. All rights reserved. SAP HANA SPS 09 - What’s New? HANA Dynamic Tiering SAP HANA Product Management November 2014 (Delta from SPS 08 to SPS 09)
Jul 02, 2015
1 © 2014 SAP SE or an SAP affiliate company. All rights reserved.
SAP HANA SPS 09 - What’s New? HANA Dynamic Tiering
SAP HANA Product Management November 2014
(Delta from SPS 08 to SPS 09)
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 2 Public
Disclaimer
This presentation outlines our general product direction and should not be relied on in making
a purchase decision. This presentation is not subject to your license agreement or any other
agreement with SAP.
SAP has no obligation to pursue any course of business outlined in this presentation or to
develop or release any functionality mentioned in this presentation. This presentation and
SAP’s strategy and possible future developments are subject to change and may be changed
by SAP at any time for any reason without notice.
This document is provided without a warranty of any kind, either express or implied, including
but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or
non-infringement. SAP assumes no responsibility for errors or omissions in this document,
except if such damages were caused by SAP intentionally or grossly negligent.
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 3 Public
Agenda
Positioning
What is “SAP HANA Dynamic Tiering”, and what is its value to the customer?
Technical Details
Implementation choices
Use Cases
SAP BW and native HANA applications
Future Direction
Where are we headed?
Positioning What is “SAP HANA Dynamic Tiering”, and what is its value to the customer?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 5 Public
IDC predictions for 2014
Data explosion Data volumes will continue to explode to 6 billion petabytes
Social networking Social networking will become embedded
in cloud platforms and most enterprise
apps and processes
Cloud Cloud spending will surge by 25%, reaching
over $100 billion. There will be a doubling of
cloud data centers.
Internet of Things 30 billion devices, sensors in 2020 – driving
$8.9 Trillion in revenue
Mobile
CRM Data
Planning
Opportunities
Transactions
Customer Sales Order
Things
Instant Messages
Demand
Inventory
Big Data
Sales
Order
Things
Mobile Demand
Big Data
CRM Data
Customer Planning Transactions
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 6 Public
SAP End to End Data Management for Real Time Business
Business & Consumer Applications
Big Data
SAP DATA MANAGEMENT
STORE TRANSACT PREDICT ANALYZE
Custom Development
ISVs & OEMs ERP
Internet of
Things
Workforce of
the Future
Cloud
Industries
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 7 Public
e
SAP HANA platform
Processing Engine
Application Function Lib. & Data Models
Integration Services
SAP HANA PLATFORM Real-time transactions + end-to-end analytics
Operational
Analytics
Big Data
Warehousing
Predictive, Spatial &
Text Analytics
REAL-TIME ANALYTICS
Sense &
Respond Planning &
Optimization
Consumer
Engagement
REAL-TIME APPLICATIONS
SAP ESP
SAP ASE
Replication
Server
SAP SQL
Anywhere
SAP IQ
SAP Data
Services
Extended Application Services
SAP Data Management Portfolio End-to End Data Management & App Platform for Real-Time Business
Database
Services SAP HANA
dynamic tiering
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 8 Public
Time Value of Data
Time
Value
Last time
accessed
Value of
immediate
data access
declines
When you
need it again
Archive Access Event
• Regulatory audit
• Business critical reference data
• Source data
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 9 Public
Multi-Temperature Storage Options with SAP HANA
Data Temperature Storage Option SAP BW
on HANA
SAP Business Suite on
HANA
SAP HANA
Native
hot SAP HANA
In-Memory
cold
SAP HANA
dynamic tiering (1) (2) Data Aging
(Next Gen ILM) 3 Near-line Storage
(NLS)
frozen Data Archiving
(ADK)
Generally available
Combination not available
1 Early shipment available for SAP BW 7.4; General availability planned Q4/2014
2 General availability with limited scope planned Q4/2014
3 For selected business objects
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 10 Public
SAP HANA Dynamic Tiering Key aspects at a glance
Add-on Product to SAP HANA
Manage data of different temperatures
Hot data (always in memory) – classical HANA
Cold data (disk based data store)
Introducing a new type of table:
Extended table – disk-based columnar table
SPS 9 release focus
Operational integration
Common installer
Unified monitoring and administration
Integrated backup/recovery
Initial functional scope
Transparent query processing
Cross-store optimizer
Use extended table in calculation views
Applications manage data temperatures
(no active support for aging)
SAP HANA Database
Data for daily reporting,
other high-priority data
Other data required to
operate the application
Hot
Warm
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 11 Public
Introducing SAP HANA Dynamic Tiering Requirements from our customers
Manage data cost effectively, yet with desired performance based on SLAs
Handle very large data sets – terabytes to petabytes
Update and query all data seamlessly via HANA tables
Application defines which data is “hot”, and which data is “warm”
Native Big Data solution to handle a large percentage of enterprise data needs without Hadoop
SAP HANA
hot store
(in-memory)
SAP HANA warm store
(dynamic tiering)
Extended table
(definition)
Extended table
(data)
Fast data movement and optimized push
down query processing
All data of extended table resides in warm store
SAP HANA Database System
Hot table
(definition/data)
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 12 Public
Hot/Warm Data Management Questions about SAP HANA Dynamic Tiering
Size and cost constraints may prohibit all in-memory solution
Not all data has the same value
Warm data has lower latency requirements than hot data
Why is warm data management important for SAP HANA?
SAP HANA dynamic tiering utilizes disk backed, smart column store technology based on Intellectual Property from SAP Sybase
SAP HANA dynamic tiering excels at ad hoc queries on structured data from terabyte to petabyte scale
SAP HANA dynamic tiering is a deeply integrated, high performance solution in a single system
Why is SAP HANA dynamic tiering the best solution for warm data
management?
Hadoop has unlimited capacity for raw data processing
Hadoop is best suited for batch processing of raw, unstructured data
Hadoop is an external data store with technical integration into HANA – with higher TCO in order to manage the additional system
What about Hadoop for warm data storage and processing?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 13 Public
SAP HANA Dynamic Tiering Key aspects at a glance
Data in the database
Different data temperatures
Maximum access performance
Hot data - always in memory
Reduced access performance:
Warm data - not (always) in memory
All part of the database’s data image
Data moved out of the database
Different data qualities
Available for read access
BW Near-line storage
Not accessible without IT process
Traditional archive
Data is stored and managed outside of the
application database SAP HANA Database
Data for daily reporting,
other high-priority data
Other data required to
operate the application
Hot
Warm
NLS Data that is (normally) not updated, infrequently accessed
Traditional Archive Data that‘s kept for legal reasons or similar
Externalize
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 14 Public
Problems with temperatures There are too many options – across system boundaries
In DB
In memory
No restrictions, all features available
External to DB
Near-line Storage
Read access, no updates
In DB
On disk
No restrictions, all features available
hot
warm
cold
???
External to DB
Archive storage
No read access or updates
Performance
and Price Priority and
Data Volume
HANA
Archive
HANA column and
row store
Warm store of dynamic tiering /
Non-Active Data Concept
BW Near-line Storage
Traditional Archive
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 15 Public
Problems with temperatures There are too many options – across system boundaries
In DB
In memory
No restrictions, all features available
External to DB
Near-line Storage
Read access, no updates
In DB
On disk
No restrictions, all features available
hot
warm
cold
???
External to DB
Archive storage
No read access or updates
Performance
and Price Priority and
Data Volume HANA column and
row store
Warm store of dynamic tiering /
Non-Active Data Concept
BW Near-line Storage
Traditional Archive
hot
warm
BW NLS
Archive
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 16 Public
SAP HANA Dynamic Tiering Map data priorities to data management
Hot Store
Classic HANA tables
Primary data image in memory
DB algorithms optimized for in-memory data
Persistence on disk to guarantee durability
Warm Store
Extended Tables
Primary data image on disk
Data processing using algorithms optimized for
disk-based data
Main memory used for caching and processing.
SAP HANA Database
Primary image in memory
Durability
Cache / Processing
Primary Image
on disk
Dynamic Tiering
Hot data
Warm data
All in one
database
Hot Store Warm Store
RAM
Technical Details Implementation choices
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 18 Public
SAP HANA Dynamic Tiering – one database / one experience for HANA application developers and admins
SAP HANA Dynamic Tiering
Reduced TCO
Optimized for performance
Single database experience
Centralized operational control
Centralized
monitoring /
admin
High speed
data ingest
Common
installer and
licensing
model
Unified
backup and
restore
Integrated
security
Optimized
query
processing
SAP
HANA
Dynamic
Tiering
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 19 Public
SAP HANA Dynamic Tiering The overall system layout
SAP HANA with Dynamic Tiering consists of two types of
hosts:
Regular worker hosts (running the classical HANA processes:
indexserver, nameserver, daemon, xsserver,…)
– HANA hosts can be single-node or scale-out; appliance or TDI
“ES host” (running nameserver, daemon, and esserver)
– esserver is the database process of the warm store
One single SAP HANA database: one SID, one instance number
All client communication happens through index server / XS server
Hot Store
Fast data movement and optimized push down query processing
SAP HANA System with dynamic tiering service
Worker
host(*)
Worker
host
Worker
host
Client
Application
Connect
ES host
Column
Table
Row
Table
Extended
Table
Warm Store
Common Storage System
(*) Standby hosts not shown
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 20 Public
Database Catalog
HANA Extended Tables
HANA Database
Warm
Store Data
HANA extended table
schema is part of HANA
database catalog
HANA extended table
data resides in warm
store
HANA extended table is a first
class database object with full
ACID compliance
Hot
Store
Table Definition
Data
Table Definition
Classical HANA
column/row table
Extended table
(warm table)
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 21 Public
High Speed Data Ingest
Import from CSV files:
IMPORT FROM CSV FILE ‘bigfile.csv’ INTO t1
Bulk array insert:
INSERT INTO t1 (col1, col2, col3...) VALUES (val1, val2, val3...)
High-speed data movement between HANA tables and HANA extended tables:
INSERT INTO t_extended select c1 FROM t_hana
Concurrent inserts from multiple connections:
A HANA extended table may be a DELTA enabled table, which allows multiple concurrent writes
Warm
Extended
Table
IMPORT FROM CSV
FILE ‘data.csv’
INTO t_extended
CSV
DATA
Hot HANA
column Table INSERT...SELECT
Materialization
Data movement between hot and warm store
HANA Database
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 22 Public
Optimized Query Processing
Parallel query processing
• Data is pulled from HANA hot store into HANA warm
store query processing engine using multiple streams,
and processed in parallel
Push/Pull query optimization and transformation
• Query operations ship to hot or warm store as
appropriate for native performance
Extended tables may be used in HANA CALC
views
• HANA Calc engine and HANA SQL engine share
extended table query performance optimizations
Joining
Grouping
Ordering
T3 T4 T1 T2
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 23 Public
Example Query Plan
select
"account_num",
count(*) as account_count
from
VXM_FOODMART.CUSTOMER C
where
"lname" >= 'Ga' and "lname" < 'Gb'
and exists
(
select *
from
VXM_IQSTORE.PRODUCT P
where
"product_id" = "customer_id"
)
group by
"account_num"
order by
"account_num";
Customer is a native
HANA table in HANA
memory
Product is a HANA
extended table in the
warm store
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 24 Public
HANA Monitoring and Administration
HANA Cockpit:
New, web based monitoring and administration
console for HANA Extended Storage
HANA Studio will be used for design and
modeling of HANA extended tables
HANA Cockpit displays status,
CPU/memory/storage resource utilization,
table usage statistics
Provides access to and search of server logs
and custom traces
Shows alerts triggered by extended storage
Enables administration of extended storage:
add and drop storage, or increase size of file
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 25 Public
Unified Backup and Restore
HANA backup manages backup of both hot and warm store
Point in Time Recovery (PITR) is supported
Extended
Storage
HANA
Data backups
(manual or
scheduled) Log backups
(automatic, or
none)
Data backup
Log backup System crash
Restore
Time
t1 t2 t3
Data backups with log
backups allow restore
to Point in Time or
most recent state: t1-
> t3
Data backups alone
allow restore to specific
backup only: t1 or t2
Log area
Backup History
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 26 Public
High Availability and Disaster Recovery
High availability
Compute node failure will result in failover to standby node (manual for warm store
nodes)
Storage failure will depend on inherent storage vendor disk mirroring and fault
tolerance capabilities
Hot and warm store should use the same storage to facilitate auto-failover in the
future
Disaster recovery
HANA without Dynamic Tiering supports continuous replication to maintain a disaster
recovery site
HANA with Dynamic Tiering will maintain a disaster recovery site through backup and
restore capabilities only
– Disaster recovery through system replication is planned for a future release
– Disaster recovery through storage replication may be added independently from
software releases Classical HANA services
Compute
node
Hot Store
Warm Store Service
Compute
node
Standby
node
Manual
Failover
Standby
node
Warm Store
Auto-
Failover
mirror
mirror
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 27 Public
Support in SAP HANA multiple database containers (MDC)
MDC: One SAP HANA system can have multiple tenant databases
Each tenant database can be associated with zero or one extended stores
Each extended store is dedicated to exactly one tenant database
SAP HANA system with MDC and dynamic tiering
Compute node
System Database
Compute node Compute node
Tenant Database <B>
Extended Store Tenant Database <A>
Tenant Database <C>
Extended Store
Classical HANA (single-node or scale-out)
ES Host <B> ES Host <C>
Use Cases SAP BW and native HANA applications
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 29 Public
SAP NetWeaver BW powered by SAP HANA Data Classification by Object Type
Frequent reporting and/or HANA-native operations
BW – Operational Data
Data Categories in a BW System
Staging Layer
Analytic Mart
Business Transformation
EDW Propagation
EDW Transformation
Co
rpo
rate
Me
mo
ry
Arc
hiv
e/N
LS
“Old”, “out-of-use” data – Archive, read-only, different SLAs
Limited reporting, limited HANA-native operations
Archived
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 30 Public
SAP HANA database
Database Catalog
Extended Tables in HANA BW Use Case: Staging and Corporate Memory
Object Classification in BW
Data Sources and write-optimized DSOs
can have the property “Extended Table”
Generated Tables are of type “Extended”
All BW standard operations supported –
no changes
Only minor temporary RAM required in HANA
InfoCubes and Regular or Advanced DSOs
Generate standard column table
Hot Store Warm store
BW System
Corporate Memory
Write-optimized DSO
Staging Area
Data Source
Table
Schema
Data
PSA Table Table
Schema
Data
Active Table
Data Mart
InfoCube
Table
Schema
Data
Fact Table
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 31 Public
SAP HANA Dynamic Tiering for Big Data
Cutting edge, in-memory platform
Transact/analyze in real-time
Native predictive, text, and spatial algorithms
Petascale extension to HANA with disk backed,
columnar database technology
Expand HANA capacity with warm/cool structured
data in HANA warm store
Tight integration between HANA hot store and HANA
warm store for optimal performance
SAP HANA with Dynamic Tiering provides native Big Data solution
Hot data
SAP HANA
Petascale, warm
structured data
HANA extended tables
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 32 Public
SAP HANA with Dynamic Tiering Native Big Data solution for a multitude of use cases
SAP HANA Dynamic Tiering for Big Data Use Cases across Industries
Airline route profitability analysis: SAP HANA analyzes revenue, variable operating costs (fuel,
landing fees...), and fixed operating costs in real time to make decisions on network, pricing, and
marketing to determine where to fly, when, and how often. All data must be analyzed in real time.
Financial services: Stock tick data streamed into SAP HANA for immediate price fluctuation
analysis and trading actions, with historical stock price data stored in HANA extended tables for
trend analysis and portfolio management.
Telecommunications: Network service data in HANA extended tables analyzed and correlated
with customer loyalty data in SAP HANA, to anticipate customer churn and initiate customer
retention response activities.
Public utilities: enterprise data stored in SAP HANA and large amounts of smart meter data
stored in HANA extended tables, to identify operational problems, and establish incentive pricing
for more efficient energy use.
Future Direction Where are we headed?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 34 Public
SAP HANA Dynamic Tiering roadmap
SAP HANA dynamic tiering available to be used by any
HANA application (if the application supports the
feature)
Common installer
Unified administration and monitoring using HANA
Cockpit
Extended Storage (ES) engine is part of HANA topology
Single authentication model
Single licensing model
Combined error log / trace handling
Integrated File-based backup/recovery, including point-in
time recovery
HANA ES host scale-out and auto-failover (HA)
Disaster Recovery (SAP HANA system replication)
Further integration with respect to backup/recovery
Hybrid extended tables with rule based automatic data
movement / aging
Optimization of communication between hot and warm
store
Further unification of DDL and DML for HANA
extended tables
Further optimizer enhancements
Further extension of unique HANA capabilities to warm
store
FUTURE PLANNED
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 35 Public
Hybrid extended tables
Automatic, rules-based, asynchronous data movement between hot and warm stores
Hot partitions in HANA memory; remaining partitions in warm store
Single HANA table that spans hot and warm stores
Hot data in
HANA tier
Warm data In
warm tier
2012 2012 Hybrid
Extended
Table aging
regulatory
audit
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 36 Public
How to find SAP HANA documentation on this topic?
• In addition to this learning material, you can find SAP HANA
platform documentation on SAP Help Portal knowledge center at
http://help.sap.com/hana_platform.
• The knowledge centers are structured according to the product
lifecycle: installation, security, administration, development:
SAP HANA Options
SAP HANA Advanced Data Processing
SAP HANA Dynamic Tiering
SAP HANA Enterprise Information Management
SAP HANA Predictive
SAP HANA Real-Time Replication
SAP HANA Smart Data Streaming
SAP HANA Spatial
• Documentation sets for SAP HANA options can be found at
http://help.sap.com/hana_options:
SAP HANA Platform SPS
What’s New – Release Notes
Installation
Administration
Development
References
•
© 2014 SAP SE or an SAP affiliate company. All rights reserved.
Thank you
Contact information
Richard Bremer, Courtney Claussen, Balaji Krishna, and Robert Waywell
SAP HANA Product Management
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 38 Public
© 2014 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate
company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its
affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services
are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an
additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or
release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future
developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for
any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-
looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place
undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.