SAP High-Performance Analytic Appliance 1.0 (SAP HANA) A First Look At The System Architecture Marc Bernard SAP Technology Regional Implementation Group February 2011
May 10, 2015
SAP High-Performance AnalyticAppliance 1.0 (SAP HANA)A First Look At The System Architecture
Marc BernardSAP Technology Regional Implementation Group
February 2011
© 2011 SAP AG. All rights reserved. / Page 2
Disclaimer
This presentation outlines our general product direction and should not be relied on inmaking a purchase decision. This presentation is not subject to your licenseagreement or any other agreement with SAP.
SAP has no obligation to pursue any course of business outlined in this presentationor to develop or release any functionality mentioned in this presentation. Thispresentation and SAP's strategy and possible future developments are subject tochange and may be changed by SAP at any time for any reason without notice.
This document is provided without a warranty of any kind, either express or implied,including but not limited to, the implied warranties of merchantability, fitness for aparticular purpose, or non-infringement. SAP assumes no responsibility for errors oromissions in this document, except if such damages were caused by SAPintentionally or grossly negligent.
© 2011 SAP AG. All rights reserved. / Page 3
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 4
ERP
Architecture OverviewIn-Memory Computing Engine and Surroundings
ERP DB
In-Memory Computing Engine
Clients (planned, e.g.) BI4 Explorer
DashboardDesign
SAP BI4 universes(WebI,...)
Request Processing / Execution Control
MS Excel
BI4 Analysis
SQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
In-Memory Computing Studio
Administration Modeling
LoadController
ReplicationAgent
ReplicationServer
SAP Business Objects BI4
DataServicesDesigner
SBO BI4servers
( programfor client)
SBO BI4InformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
© 2011 SAP AG. All rights reserved. / Page 5
ERP
Architecture OverviewThe Engine
LogERP DB
Clients (planned, e.g.) SBOP Explorer 4.0
Xcelsius SAP BI universes (WebI,...)
MS Excel
SBOP Analysis
IMC Studio
Administration Modeling
LoadController
ReplicationAgent
Business Objects Enterprise
DataServicesDesigner
SBO serverprogramsfor clients
SBOInformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
ReplicationServer
© 2011 SAP AG. All rights reserved. / Page 6
ERP
Architecture OverviewLoading Data into SAP HANA
ERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
In-Memory Computing Studio
Administration Modeling
LoadController
ReplicationAgent
ReplicationServer
Business Objects Enterprise
DataServicesDesigner
SBO BI4servers
( programfor client)
SBOInformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
Clients (planned, e.g.) BI4 Explorer
DashboardDesign
SAP BI4 universes(WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 7
ERP
Architecture OverviewData Modeling
ERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
In-Memory Computing Studio
Administration Modeling
LoadController
ReplicationAgent
ReplicationServer
Business Objects Enterprise
DataServicesDesigner
SBO BI4servers
( programfor client)
SBOInformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
Clients (planned, e.g.) BI4 Explorer
DashboardDesign
SAP BI4 universes(WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 8
Clients (planned, e.g.)
ERP
Architecture OverviewReporting
ERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
In-Memory Computing Studio
Administration Modeling
LoadController
ReplicationAgent
ReplicationServer
Business Objects Enterprise
DataServicesDesigner
SBO BI4servers
( programfor client)
SBOInformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
BI4 Explorer
DashboardDesign
SAP BI4 universes(WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 9
ERP
Architecture OverviewAdministration
ERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
TransactionManager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
AuthorizationManager
MetadataManager
In-Memory Computing Studio
Administration Modeling
LoadController
ReplicationAgent
ReplicationServer
Business Objects Enterprise
DataServicesDesigner
SBO BI4servers
( programfor client)
SBOInformationDesign Tool
Other Source Systems
SAPNetWeaver
BW3rd Party
DataServices
Clients (planned, e.g.) BI4 Explorer
DashboardDesign
SAP BI4 universes(WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 10
DB Server
SAP High-Performance Analytic Appliance 1.0
SAP HANA
JDBC ODBC ODBO SQLDBC
SAP In-MemoryComputing Engine
ReplicationServer
SAP In-Memory Computing Studio
SAP BusinessApplication
ReplicationAgent
SAP BusinessObjectsData Services 4.0
Anysource
SAPBusinessObjects
BI 4.0
Repository
SAP BusinessObjects BI clients
SQ
L
MD
X
BIC
S
Auth
entic
atio
nC
onte
nt m
gmt
sync
Adm
in &
mod
el
load (optional)
(optional)
(optional)
(existing)
© 2011 SAP AG. All rights reserved. / Page 11
Request Processing and Execution ControlConceptual View
Standard SQLProcessed directly by DB engine
SQL Script, MDX and planning engineinterface
Domain-specific programminglanguages or modelsConverted into calculation models
Calc EngineCreate logical execution plan forcalculation modelsExecute user defined functions
Relational EngineDB optimizer produces physicalexecuting planAccess to row and column store
© 2011 SAP AG. All rights reserved. / Page 12
Calc Engine for Dummies
The easiest way to think of Calculation Models is to see them as dataflow graphs,where the modeler can define data sources as inputs and different operations (join,aggregation, projection,…) on top of them for data manipulations.
The Calculation Engine will break up a model, for example some SQL Script, intooperations that can be processed in parallel (rule based model optimizer). Then theseoperations will be passed to the database optimizer which will determine the bestplan for accessing row or column stores (algebraic transformations and cost basedoptimizations based on database statistics).
© 2011 SAP AG. All rights reserved. / Page 13
Calc Engine for DummiesExample
© 2011 SAP AG. All rights reserved. / Page 14
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 15
In-Memory Computing EngineHigh Level Architecture
Row StoreOne of therelational enginesInterfaced fromcalculation /execution layerPure in-memorystore
Persistencemanaged inpersistencelayer
SAP in-memorycomputing engine
HANA
© 2011 SAP AG. All rights reserved. / Page 16
Row Store ArchitectureRow Store Block Diagram
Row Store Block DiagramTransactional Version Memory
Contains temporary versionsNeeded for Multi-VersionConcurrency Control (MVCC)
SegmentsContain the actual data (content ofrow-store tables) in pages
Page ManagerMemory allocationKeeping track of free/used pages
Version Memory ConsolidationThink ‘garbage collector for MVCC’
Persistence LayerInvoked in write operations (log)And in performing savepointscheckpoint writer
© 2011 SAP AG. All rights reserved. / Page 17
Row Store ArchitectureHighlights
Write OperationsMainly go into “Transactional VersionMemory”“INSERT” also writes to PersistedSegment
Read Operations
Write Operations
TransactionalVersionMemory
Main Memory
PersistedSegment
Data thatmay be
seen by allactive
transactions
Recentversions ofchangedrecords
Version MemoryConsolidation
Version ConsolidationMoves “visible version”from Transaction VersionMemory into PersistedSegment (based onCommit ID)Clears “outdated” recordversions from TransactionalVersion Memory
Memory HandlingRow store tables arelinked list of memorypagesPages are grouped insegmentsPage size: 16 KB
Persisted SegmentContains data that may be seen by anyongoing transactionData that has been committed beforeany active transaction was started)
© 2011 SAP AG. All rights reserved. / Page 18
Indexes for Row Store TablesPrimary Index / Row ID / Index Persistence
Each row-store table has a primary indexPrimary index maps ROW ID primary key of table
ROW ID: a number specifying for each record its memory segment and page
How to find the memory page for a table record?A structure called “ROW ID” contains the segment and the page for the recordThe page can then be searched for the records based on primary keyROW ID is part of the primary index of the table
Secondary indexes can be created if needed
Persistence of indexes in row storeIndexes in row store only exist in memory
No persistence of index dataIndex definition stored with table metadataIndexes filled on-the-fly when system loads tables into memory on system start-up
© 2011 SAP AG. All rights reserved. / Page 19
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 20
In-Memory Computing EngineHigh Level Architecture
Column StoreOne of the relationalenginesInterfaced fromcalculation / executionlayerPure in-memory store
Persistencemanaged inpersistence layer
Optimized for highperformance of readoperationGood performance ofwrite operationsEfficient datacompression
SAP in-memorycomputing engine
HANA
© 2011 SAP AG. All rights reserved. / Page 21
Column Store ArchitectureColumn Store Block Diagram
Column Store Block DiagramOptimizer and Executor
Handles queries andexecution plan
Main and Delta StorageCompressed data for fast readDelta data for fast writeAsynchronous delta merge
Consistent View Manager
Transaction Manager
Persistence Layer
© 2011 SAP AG. All rights reserved. / Page 22
Column StoreHighlights
Storage Separation (Main & Delta)Enables high compression and high writeperformance at the same time
Delta Merge OperationSee next slide
Read Operations
WriteOperations
Main
Main Memory
Delta
Writeoptimized
Compressedand
Readoptimized
Read OperationsAlways have to read from bothmain & delta storages and mergethe results.Engine uses multi versionconcurrency control (MVCC) toensure consistent read operations.
Data Compression in MainStorage
Compression by creatingdictionary and applying furthercompression methodsSpeed up
Data load into CPU cacheEquality check Search
The compression is computedduring delta merge operation.
Write OperationsOnly in delta storage because write optimized.The update is performed by inserting a newentry into the delta storage.
© 2011 SAP AG. All rights reserved. / Page 23
Column StoreDelta Management
Delta Merge OperationPurpose
To move changes in delta storage into the compressed and read optimized main storageCharacteristics
Happens asynchronouslyEven during merge operation the columnar table will be still available for read and writeoperationsTo fulfil this requirement, a second delta and main storage are used internally
Read Operations
WriteOperations
Main
Before Merge
Delta
Read Operations
WriteOperations
MainNew
After Merge
DeltaNew
Read Operations
WriteOperations
Main
During Merge
MainNew
DeltaNewDelta
Merge Operations
© 2011 SAP AG. All rights reserved. / Page 24
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 25
Persistence LayerPurpose and Scope
Why Does An In-memory Database Need A Persistence Layer?Main Memory is volatile. What happens upon…
Database restart?Power outage?...
Data needs to be stored in a non-volatile wayBackup and restore
SAP in-memory computing engine offers one persistence layer which is used by row store andcolumn store
Regular “savepoints”full persisted image of DB at time of savepoint
Logs capturing all DB transactions since last savepoint (redo logs and undo logs written)restore DB from latest savepoint onwards
Ability to create "snapshots"used for backups
© 2011 SAP AG. All rights reserved. / Page 26
Persistence LayerSystem Restart and Population of In-memory Stores
Actions During System RestartLast savepoint must be restored plus…
Undo logs must be read for uncommitted transactions saved with last savepointRedo logs for committed transactions since last savepoint
Complete content of row store is loaded into memory
Column store tables may be marked for preload or notOnly tables marked for preloadare loaded into memory duringstartupIf table is marked for loadingon demand, the restoreprocedure is invoked on firstaccess
© 2011 SAP AG. All rights reserved. / Page 27
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 28
Row Store vs. Column StoreWhen to Use Which Store
Modeling Only Possible For Column TablesThis answers the frequently asked question:"Where should I put a table – row store or column store?"
Information Modeler only works with column tablesReplication server creates tables in column store per defaultData Services creates tables in column store per defaultSQL to create column table: "CREATE COLUMN TABLE ..."Store can be changed with "ALTER TABLE …"
System Tables Are Created Where They Fit BestAdministrative tables in row store:
Schema SYS caches, administrative tables of engineTables from statistics server
Administrative tables in column store:Schema _SYS_BI metadata of created views + master data for MDXSchema _SYS_BIC some generated tables for MDXSchema _SYS_REPO e.g. lists of active/modified versions of models
© 2011 SAP AG. All rights reserved. / Page 29
SAP In-Memory Computing StudioLook and Feel
NavigatorView
Quick LaunchView
PropertiesView
© 2011 SAP AG. All rights reserved. / Page 30
SAP In-Memory Computing StudioFeatures
Information Modeler FeaturesModeling
No materialized aggregatesDatabase viewsChoice to publish and consume at 4 levels of modeling
Attribute View, Analytic View, Analytic View enhanced with Attribute View, Calculation View
Data PreviewPhysical tablesInformation Models
Import/ExportModelsData Source schemas (metadata) – mass and selective loadLandscapes
Data Provisioning for SAP Business Applications (both initial load and replication)Analytic Privileges / Security
© 2011 SAP AG. All rights reserved. / Page 31
Modeling Process Flow
Import SourceSystemmetadata• Physical tables
are createddynamically (1:1schema definitionof source systemtables)
ProvisionData• Physical tables
are loaded withcontent.
CreateInformationModels• Database Views
are created• Attribute Views• Analytic Views• Calculation
Views
Deploy• Column views
are created andactivated
Consume• Consume with
choice of clienttools
• BICS, SQL, MDX
© 2011 SAP AG. All rights reserved. / Page 32
SAP In-Memory Computing StudioTerminology
Information Modeler TerminologyData
Attributes – descriptive data (known as Characteristics SAP BW terminology)Measures – data that can be quantified and calculated (known as key figures in SAP BW)
ViewsAttribute Views – i.e. dimensionsAnalytic Views – i.e. cubesCalculation Views – similar to virtual provider with services concept in BW
HierarchiesLeveled – based on multiple attributesParent-child hierarchy
Analytic Privilege – security object
© 2011 SAP AG. All rights reserved. / Page 33
SAP In-Memory Computing StudioNavigator View - Default Catalog
HANA Instance (<USER>)
HANA Server Nameand Instance Number
User Database schema
Schema Content:Column Views,Functions, Tables,Views
© 2011 SAP AG. All rights reserved. / Page 34
SAP In-Memory Computing StudioNavigator View - Information Models
Information Models organizedin Packages
Attribute Views, Analytic Views,Calculation Views, Analytic Privilegesorganised in folders
© 2011 SAP AG. All rights reserved. / Page 35
Attribute Views
Attribute ViewWhat is an Attribute View?
Attributes add context to data.Attributes are modeled using Attribute Views.Can be regarded as Master Data tablesCan be linked to fact tables in Analytical ViewsA measure e.g. weight can be defined as an attribute.
Table Joins and PropertiesJoin Types
leftOuter, rightOuter,fullOuter, textTable
Cardinality1:1N:11:N
Language Column
© 2011 SAP AG. All rights reserved. / Page 36
Analytical View
Analytical ViewAn Analytical View can be regarded as a “cube”.Analytical Views does not store any data. The data is stored in column store or table viewbased on the Analytical View Structure.Attribute and Measures
Can create Attribute FiltersMust have at least one AttributeMust have at least one MeasureCan create Restricted MeasuresCan create Calculated MeasuresCan rename Attribute andMeasures on the property tab
© 2011 SAP AG. All rights reserved. / Page 37
Analytical View
Analytical View: Data PreviewThere are three main views one can select from when previewing data.
Raw Data – table format of dataDistinct Values – graphical and text format identifying unique valuesAnalysis – select fields (attributes and measures) to display in graphical format.
© 2011 SAP AG. All rights reserved. / Page 38
Calculation View (Scripting)
Calculation ViewDefine Table Output StructureWrite SQL Statement.
Ensure that the selected fields corresponds to previously defined Output table structure of the function.Example :SQL_A = SELECT MATNR, KUNNR, …. FROM<COPA_ACTUAL_ANALYTICAL VIEW 1>
SQL_P = SELECT MATTNR_KUNNR, … FROM<COPA_PROJECTED_ANALYTICAL VIEW 2>
TABLE_OUTPUT_STRUCTURE =SELECT * FROM <SQL_A> UNIONSELECT * FROM <SQL_P>;
© 2011 SAP AG. All rights reserved. / Page 39
SAP In-Memory Computing StudioPre-Delivered Administration Console
NavigatorView
PropertiesView
AdministrationView
© 2011 SAP AG. All rights reserved. / Page 40
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer5. Modeling6. Q&A
© 2011 SAP AG. All rights reserved. / Page 41
Thank you!
© 2011 SAP AG. All rights reserved. / Page 42
Further Information onSAP HANA and In-Memory Technologies
In-Memory Computinghttp://www.sap.com/platform/in-memory-computing
Real-Real Time Business with HANAhttp://www.youtube.com/watch?v=uUqtUw-m7mQ
SAP Community Network Topic Pagehttp://www.sdn.sap.com/irj/sdn/in-memory
SAP Community Forumhttp://forums.sdn.sap.com/forum.jspa?forumID=491
The SAP NetWeaver BW – SAP HANA Relationshiphttp://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/21575
SAP HANA Ramp-Up Knowledge Transfer (login required)http://service.sap.com/rkt-hana
SAP HANA Documentation (login required during ramp-up)https://cw.sdn.sap.com/cw/community/docupedia/hana
© 2011 SAP AG. All rights reserved. / Page 43
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained hereinmay be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries,eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+,POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex,MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or othercountries.Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.Java is a registered trademark of Sun Microsystems, Inc.JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, Clear Enterprise, SAP BusinessObjects Explorer and other SAP products and services mentioned herein as wellas their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products andservices mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP France in the United States and in other countries.All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only.National product specifications may vary.The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without theexpress prior written permission of SAP AG.This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies,developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/ordevelopment. Please note that this document is subject to change and may be changed by SAP at any time without notice.SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or otheritems contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of thesematerials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot linkscontained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.
© 2011 SAP AG. All Rights Reserved