Nearline storage - SAP BI Netweaver

IM302 Near-line Storage With SAP NetWeaver BI

Glen Leslie, Director - SAP NetWeaver RIG Americas

Dr. Peter Zimmerer, Development Architect - SAP NetWeaver BI

© SAP 2008 / SAP TechEd 08 / <IM302> Page 2

Disclaimer

This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.


Session IM302 Abstract

Near-line Storage With SAP NetWeaver BI

In the context of information lifecycle management, SAP NetWeaver 7.0 offers a near-line storage (NLS) interface that allows to export operational data from the database into NLS partitions under control of a particular NLS vendor solution. Several vendors are already certified for this interface. NLS partitions of an InfoProvider are read-only and therefore have to be seen as a type of archive. Nevertheless, from the query and DTP perspective the existence of NLS partitions is transparent. The NLS concept establishes scalability for database volumes, reduces TCO for database management and administration, and complements with the SAP NetWeaver Business Intelligence Accelerator (SAP NetWeaver BI Accelerator), keeping old data - that normalynobody wants to look at - away from the valuable SAP NetWeaver BI Accelerator memory space. Finally, NLS support is offered for write-optimized DataStore Objects (7.0 EHP1) and for MultiProviders (7.2) as well. The session reflects the entire NLS concepts as well as the latest enhancements and optimizations discussing various scenarios with the help of numerous system demonstrations.


1. Concept of Nearline Storage1.1. Basic ideas of cross media storage managment1.2. Information Lifecycle Management

2. Nearline Solution of SAP NetWeaver BI2.1. Architecture2.2. Implementation Steps

3. Latest Enhancements3.1. NLS for write-optimized DataSore Objects and MultiProvider3.2. Look-up API 3.3. Administration Improvements

4. Summary and Outlook4.1. Roadmap and time line

Agenda


Information as Corporate Asset – We Do not Know What we not Know...

The Known Current BI implementations are set up

to answer known requirements

The UnknownLittle or nothing is done to be prepared for

unpredictable future information needs


DSS Applications Departmental Data Marts

EDW

MarketingAcctg Finance

Sales ERPERP

ERP

CRM

eComm.

Bus. Int.

ETL

GlobalODS

Oper.Mart

Exploration warehouse/ data mining

Stag

ing

Are

a

localODS

DialogueManager

CookieCognition

Preformatteddialogues

Cross mediaStorage Management

NearlineStorage

Web Logs

SessionAnalysis

Internet

ERPCorporate

Applications

ChangedData

GranularityManager

Archives

Bill Inmon’s: Enterprise Data Warehousing Concept

Source:Bill Inmon


Bill Inmon’s Data-Aging Strategies

Online Database Near-Line Storage Classic Archive

Frequently read / changed data

(actual)

Infrequently read data (mature)

Very rarely read data (aged)

Information Lifecycle according to Importance/Age:

Presenter

Presentation Notes

As the access frequency for data is decreasing with its age, data has to be moved to cheaper devices the older it gets.


Classic Archiving in BW vs. NLS in SAP NW 7.0 BI

Access Frequency/Possibility

Age of Data

Archiving (SAP BW 3.X)ADK-based (Archive Development Kit) archiving solution for InfoCubes and ODS objectsCost reduction using alternative storage mediaArchived data must be reloaded into the SAP NetWeaver BI database for analysis purposes

Online Archive ArchiveReloadOnline

Online Near Line Storage

NLS (SAP NW 7.0 BI)SAP NetWeaver BI has direct access to NLS data Availability of historic data for reporting and propagation with reduced costsReload of NLS-based data into the InfoCube or DataStore Object notnecessary for analysis purposes






Agenda


Analytic Engine

BI Architecture: Platform & Data Warehouse

Dat

a Fl

ow C

ontr

ol /

Proc

ess

Cha

ins

Business Explorer Suite (BEx)

Caching

Data Acquisition

Mon

itorin

g / A

dmin

istr

atio

n

Met

a D

ata

Rep

osito

ry /

Doc

umen

ts

Calculation

Aggregation

Planning Services

Planning Functions

andSequences

Data Warehouse

Operational Data Store(volatile) Data Warehouse Layer

(historical)

Architected Data Marts

Open Hub

Service

DataSource PSA

Analysis Process Design

BI A

ccel

erat

or E

ngin

e

Master Data

Archiving Service

Arc

hive

/ N

LS E

ngin

e


Focusing on Archiving, NLS and Business Intelligence

Main ideas of the SAP NW 7.0 BI Nearline Storage Interface:

Data Archiving Processes (DAP) result in partitioning of InfoCube and DSO dataNearline Storage fills the gap between online storage and offline storage NLS data is directly accessable via SQL-like interfaces, but read only NLS data can be highly compressedNLS data is self-contained with respect to indexing DAP leads to Archiving (ADK) or NLS or both

Technical details depending on Partner Solutions residing on top of the interface

ERPBI

ETL/EAI

ADK+ASADK

ILM

Archive NLS NLS

DAP


NLS Interface – Architectural Overview


NLS Interface – Objects, Processes and Functions

RDBMS

InfoProvider

NLSPartner Solution

BI

DB

DAP – Design Time

DAP – Run Time

define partitioning citeriagenerate DAP

extract(DAP Menu,Process Chain)restore (DAP menu)

DAP – Scheduling /Monitoring

describe partition to be extractedgenerate request InfoProviderManage MenuProcess Chain

Data Archiving Process (DAP)

Data Archiving Process (DAP)

NLS - Interface

Add. Features Query access to NLSDTP access to NLS

Presenter

Presentation Notes

The connection of a BI system to a Nearline Service is modeled by three ABAP Objects interfaces, which are designed to handle a connection to a Nearline Service for data definition, data manipulation, and data query commands, a write interface for packaged data write to a table abstraction within the Nearline Storage, and a cursor interface for packaged data read from a table abstraction within the Nearline Storage.


The Near-Line Storage Solution for SAP NetWeaver BI

Separation of frequently used data and rarelyused data using Data Archiving Processes (DAP)

Data Archiving Processes guarantee data consistency over DBMS and file system boundaries using

automatic deletes write protection for removed areas in InfoCubes and DataStore ObjectsReload option

NLS support for InfoCubes and DataStore ObjectsNLS Partions of an InfoCube are not accesseble in an MultiProvider scenario(available with the next 7.2 release)

Transparent access to NLS data forQueriesData Transfer Processes, DAP Restore

Open interface for partner solutions

Present development partners Certified for SAP NW 7.0 (in the sequence of certification)

SAND Technology – Dynamic Nearline Access® yesPBS Software – CBW® yesFileTek – StorHouse® yes


Offline Archive

RDBMS

Design Aspects – Near-line Storage vs. BI Accelerator

InfoMarts (InfoCube)

Near-line StorageBIA

Acquisition

Acceleration Archiving

BI

Access - very frequently frequently not frequently rarely

NLS alternative storage types with direct accesscapabilities for reporting and loadingextracted partitions are deleted in RDBMS (NLS storage and Online Storage togetherconsistently reflect the BI data persistencyof an InfoProvider)NLS data is read-only NLS part of an InfoProvider are write-protectedNLS is primarily partitioned by time/age

BIA Replication of the BI Star Schema includingmaster dataDB volume not affectedRoll-Up and Change Run possible after data loads optimized for fast BI Query access (loadbalancing by hash partitioning)






Agenda


Overview About Latest Enhancements

Archiving of write-optimized DataStore ObjectsSupport of Data Acquisition Layer ( Corporate Memory)

Nearline access through MultiProviderDecoupling of reporting layer and persistency layer

Look up of Nearline data (e.g. for staging processes)Support of Data Integration Layer

Miscellaneous enhancementsEnhanced status and job monitoring within InfoProvider management viewSuspension and selective continuation of archiving processes within Process ChainsRestore of an archiving request with all successorsSmaller Data Object size for ADK-based Nearline Solution without semantic grouping


Archiving of Write-Optimized DataStore – Properties and Modeling Recommendations

New DataStore type "Write-optimized DataStore Object" introduced with NW 7.0Only active data table (no ChangeLog, no ActivationQueue)Generated technical key (Request Id, Packet No, Record No)

mass insert into active data tableno activation step necessary

Additional semantic keyMust be defined, butUniqueness check can be switched off

No SID generation (but reporting is possible)Fully integrated into the staging process (can be used as source and target as well)

Recommended to be used within the Data Acquisition Layer as "Pass Thru"Finest granularity of dataHighest data volumeBest place to be prepared for the unforeseen ( Corporate Memory)

IssueNot archivable with SAP NetWeaver BI 7.0


Archiving of Write-Optimized DataStore – Request-Based Archiving

New obligatory Selection Profile "Request-Based Archiving"

Relative time selection will be defined in order to derive a "Less than" selection for Request IDEither based on technical request attribute creation date or loading date or …… absolute time characteristic from data

Archiving takes place with strictly increasing Request IDConsequently restore must take place with strictly with decreasing Request ID

Selection Profile depends on DSO property "Do Not Check Uniqueness of Data"Property is not set (= default)

An absolute time characteristic or a time-correlated partitioning characteristic from the semantic key must be chosen in order to support check of uniqueness by range locks

Property is set (i.e. uniqueness needs not to be checked)Get minimum request id rmin having creation/loading date greater than threshold date dnArchive all requests having request Id R < rmin

Currently being loaded

Currently being archived/restoredArchived


Archiving of Write-Optimized DataStore – Variants of Request-Based Selection Profile

Use creation/loading date as timeslice

char.

Archive all requests having request Id R ≥

rn-1 and R < rn (with r0 = 0)

Calculate threshold date dn from current date, time shift, and rounding unit

Convert threshold date dn to value of timeslice characteristic tn

Get minimum request Id rn having timeslice characteristic

value greater than tn

Get minimum request Id rn having loading/creation

date greater than dn

Check uniquess

of data

No

Yes No

Timeslice characteristic is part

of the semantic key

Yes

Convert threshold date dn to value of timeslice characteristic tn

pn := tn

Get minimum request Id rn having partitioning characteristic

value greater than pn

No

Get minimum value pn for partitioning characteristic

having timeslice characteristic value greater than tn

Lock range P ≥

pn and P < pn

Yes

Should be preferred

Fastest

Slowest


Archiving of write-optimized DataStore – Demo

Create a Data Archiving ProcessCreate and schedule archiving requestsRestore archiving requestsLoad data to subsequent Data Targets

LSA

Data Propagation Layer Corporate Memory

Data Acquisition Layer

Data Acquisition Layer

DataSource

InfoSource

InfoPackage

InfoPackage

RDA-DTP

Nearline Storage

Reporting Layer (Architected Data Marts)Reporting Layer (Architected Data Marts)

MultiProviderSAP Sales InfoCube

Propagating DataStoreStandard DSO

DAP

DTP DTP

DTP

PSA

RDA-Daemon (Pull ~ 1/min)

DTP

1

23

4

1234

Corporate Memory DSOWrite optimized DSO








Nearline Access through MultiProvider

Decoupling of Reporting Layer from Persistency Layers

Recommendation:

Usage of MultiProviders

Issue in NW 7.0:

MultiProvider does not support access of Nearline partitions of its Part Providers

Solution with NW 7.2:

Support enabled with NW 7.2Development already finishedTests are ongoing

Availability:

Ramp-Up to customer (RTC) planned for Q4/2009

Queries

MultiProvider

Nearline Storage








NLS

EMPL_ID DATE_TO DATE_FROM … SALARY …4711 31.12.2007 01.01.2007 … 50 000 …4712 31.12.2007 01.01.2007 … 62 000 …4711 31.12.2008 01.01.2008 … 51 000 …4712 31.12.2008 01.01.2008 … 64 000 ...

Look up of Nearline Data

EMPL_ID SDATE … SALARY …4711 27.12.2007 … 50 000 …4711 26.01.2008 … 51 000 …4712 27.12.2007 … 62 000 …

EMPL_ID SDATE …4711 27.12.2007 …4711 26.01.2008 …4712 27.12.2007 …

Target

Source

Lookup DSO

Target

Source

Lookup DSO

L.EMPL_ID = S.EMPL_ID AND L.DATE_TO >= S.SDATE AND L.DATE_FROM <= S.SDATE


Look up – without Nearline Storage Code Snippet from Transformation ... *$*$ begin of global - insert your declaration only below this line *-* ... DATA pt_lookup_data TYPE SORTED TABLE OF ... WITH UNIQUE KEY emplid date_to. ... *$*$ end of global - insert your declaration only before this line *-* ... METHOD start_routine. ... *$*$ begin of routine - insert your code only below this line *-* ... * collect keys for lookup from data package LOOP AT source_package ASSIGNING <source_fields>. MOVE-CORRESPONDING <source_fields> TO ls_lookup_key. APPEND ls_lookup_key TO lt_lookup_keys. ENDLOOP. * lookup from DSO table SELECT emplid date_to date_from salary INTO TABLE pt_lookup_data FROM (l_tabname) FOR ALL ENTRIES IN lt_lookup_keys WHERE emplid = lt_lookup_keys-emplid AND dateto >= lt_lookup_keys-sdate AND datefrom <= lt_lookup_keys-sdate. ... *$*$ end of routine - insert your code only before this line *-* ENDMETHOD. "start_routine ...


Look up – with Nearline Storage Adjustments to Transformation

SELECT emplid date_to date_from salary INTO TABLE pt_lookup_data FROM (l_tabname) FOR ALL ENTRIES IN lt_lookup_keys WHERE emplid = lt_lookup_keys-emplid AND dateto >= lt_lookup_keys-sdate AND datefrom <= lt_lookup_keys-sdate.

DATA: lt_lookup_conditions TYPE cl_rsda_infoprov_query=>t_entry_fields, ls_lookup_condition LIKE LINE OF lt_lookup_conditions. ls_lookup_condition-fieldname = 'EMPL_ID'. ls_lookup_condition-option = 'EQ'. ”= ls_lookup_condition-component_name = 'EMPL_ID'. APPEND ls_lookup_condition TO lt_lookup_conditions. ls_lookup_condition-fieldname = 'DATE_TO'. ls_lookup_condition-option = 'GE'. ”>= ls_lookup_condition-component_name = 'SDATE'. APPEND ls_lookup_condition TO lt_lookup_conditions. ls_lookup_condition-fieldname = 'DATE_FROM'. ls_lookup_condition-option = 'LE'. ”<= ls_lookup_condition-component_name = 'SDATE'. APPEND ls_lookup_condition TO lt_lookup_conditions. TRY. CALL METHOD cl_rsda_infoprov_query=>select EXPORTING i_infoprov = c_infoprov i_t_entries = lt_lookup_keys i_t_entry_fields = lt_lookup_conditions CHANGING c_t_data = pt_lookup_data. CATCH cx_rsda_input_invalid cx_rsda_access_error. * Do error handling ... ENDTRY.

Replace

SELECT … FOR ALL ENTRIES …

by

API call


Look up of Nearline Data

Available with SAP NetWeaver BI 7.0 Support Package 10

See SAP note 985609 and related notes for details

Extension to Nearline Interface as prototype for pilot customers availableto solve performance problems with temporal lookups


Miscellaneous Enhancements

Enhanced status and job monitoring within InfoProvider management viewIncludes monitoring of archive instances within process chains and ADK-based jobsIndicator for “active” archiving requestsAvailable with SAP NetWeaver BI 7.0 Enhancement Package 1

Suspension and selective continuation of archiving processes within Process Chains

Option: Continue only archiving requests created by predecessor instancesAvailable with SAP NetWeaver BI 7.0 Support Package 18

Restore of an archiving request with all successorsAvailable with SAP NetWeaver BI 7.0 Support Package 18

Smaller Data Object size for ADK-based Nearline Solution without semantic grouping

Data Object size reduced from 20 MByte to 1 MByteImproves performance for ADK-based Nearline Solutions like PBS CBWAvailable with SAP NetWeaver BI 7.0 Support Package 19 and Enhancement Package 1






Agenda


Summary

General aspectsNLS fills the gap between online storage and offline storage with its data residing neither in the BI data base nor in a classical archiveNLS data can be accessed directly for analyses and data load purposes (BI Queries, DTP’s, DAP Restore Feature)NLS handling is provided for InfoCubes and DataStore Objects and processed using Data Archiving Processes (DAP)NLS data is read-only and thus promotes cost-efficient storage management

Vendor specific aspectsNLS data can be compressed (up to 95%)NLS data is intrinsically indexedNLS data can reside on a separate database or file system


Outlook

Medium termEnhanced lookup support especially for temporal lookups (non-equal lookup conditions)Request-based archiving for InfoCubes (avoid compression before archiving)Combined DTP extraction from online and archive partition of an InfoCube

Long termArchiving of InfoCubes with non-cumulative key figuresArchiving of master data and hierarchiesNLS support for InfoSets…


SDN Subscriptions offers developers and consultants like you, an annual license to the complete SAP NetWeaver platform software, related services, and educational content, to keep you at the top of your profession.

SDN Software Subscriptions: (currently available in U.S. and Germany)A one year low cost, development, test, and commercialization license to the complete SAP NetWeaver software platform Automatic notification for patches and updatesContinuous learning presentations and demos to build expertise in each of the SAP NetWeaver platform componentsA personal SAP namespace

SAP NetWeaver Content Subscription: (available globally)An online library of continuous learning content to help build skills.

Starter Kit

Building Your Business with SDN Subscriptions

To learn more or to get your own SDN Subscription, visit us at the Community Clubhouse or at www.sdn.sap.com/irj/sdn/subscriptions

http://www.sdn.sap.com/irj/sdn/subscriptions


Further Information

Related Workshops/Lectures at SAP TechEd 2008IM362 Enterprise Data Warehousing With SAP NetWeaver BI:

Designing an Effective Corporate Memory, H2

Related SAP Education and Certification Opportunitieshttp://www.sap.com/services/education/catalog/netweaver/bi.epxservice.sap.com/okp

SAP Public Web:SAP Developer Network (SDN): https://www.sdn.sap.com/irj/sdn/nw-biBusiness Process Expert (BPX) Community: www.bpx.sap.com

http://www.sap.com/services/education/catalog/netweaver/bi.epx

https://www.sdn.sap.com/irj/sdn/nw-bi

http://www.bpx.sap.com/


Thank you!


Please complete your session evaluation.Be courteous — deposit your trash,

and do not take the handouts for the following session.

Thank You !

Feedback

Nearline storage - SAP BI Netweaver

Documents