Unlocking the secrets to how essbase thinks e roske in sync10 oracle epm track

Unlocking the Secrets to

How Essbase Thinks

Edward Roske

[email protected]

BLOG: LookSmarter.blogspot.com

WEBSITE: www.interrel.com

TWITTER: ERoske

About interRel

• 2008 & 2009 Oracle Titan Award winner - EPM Solution of the

year

• 2008 Oracle EPM Excellence Award

• 2009 Oracle EPM/BI Innovation Award

• One of the fastest growing companies in the world

(Inc. Magazine, ‘08 & ‗09)

• Two of the four Hyperion Oracle ACE Directors in the world

• Oracle Gold Specialized Partner in Oracle EPM

• Focused exclusively on Oracle Hyperion EPM software

– Consulting

– Training

– Infrastructure and Installation

– Support

– Software sales

• 5 Hyperion Books Available:

– Essbase (7): Complete Guide

– Essbase System 9: Complete Guide

– Essbase System 9: End User Guide

– Smart View 11: End User Guide

– Essbase 11: Admin Guide

– Hyperion Planning for End Users

• Coming in March

– Hyperion Planning for Administrators

• To order, check out www.LuLu.com

Copyright © 2007, Hyperion. All rights reserved.3

Agenda

• Introduction

• Internal Workings of Essbase: BSO

• Internal Workings of Essbase: ASO

• Question and Answer

Internal Workings of Essbase: Block Storage

Bob Earle Invented ―Sparse and Dense‖

• How is data distributed?

X X

X X

X X

X

X X

X

X

X X X

X

X

X

X

X

X

Time Periods

DENSE

Accounts

X

X

X

X

X

Products

SPARSE

Locations

Block Structure

Jan Feb Mar Q1 Q2 Q3 YrSales

COGs

Rev

Salary

Marketing

OtherExp

Expense

Profit

Headcnt

UnitsSold

Act

BudVarVar%

Period

(dense)

Account

(dense)

Data Cells Within the Block

Actual -> Profit -> Jan

Cross-dimensional operator (means Actual BY Profit BY Jan)

Blocks

• An individual block is created for each combination of sparse

stored members

Cola->New York Cola->MassachusettsCola->Florida

Cola->East

Storage and Compression

Storage

• PAG File

– Contains the data (the blocks with a header for each)

– Contains up to 2 Gb of data in each PAG file (32-bit)

– Can be 1,024 different files

– Can be compressed and fragmented

– Can be stored on multiple drive locations

• IND File

– Contains the list or pointers to the blocks (intersection of sparse dimensions)

– Contains up to 2 Gb of index in each IND file (32-bit)

– Can be 1,024 different files

– Can be fragmented

– Can be stored on multiple drive locations

Compression

• ―Simple‖ compression settings

– None

– zLib

– Index Value Pair

• Can‘t assign directly

• Good for really large blocks with really sparse data

• Following types use multiple compressions (one per block)

– Bitmap

• Good for non-repeating data

• Will use Bitmap or IVP– RLE = Run Length Encoding

• Good for data with zeros and data that repeats (such as

budgeting)

• Will use RLE, Bitmap, or IVP

Dimension Order & Compression

• Dimension order affects compression

• First dense dimension determines your ―columns‖ in PAG file

• Compression is from left to right, top to bottom

• Move Time to first dense dimension and we get:

• Notice repeating values

• Time should be dense then Measures for RLE compression

BUDGET Jan Feb Mar Apr May Jun

Sales 100 100 100 120 120 120

COGS 50 50 50 50 50 50

Margin 50 50 50 70 70 70

Exp. 30 30 30 30 30 30

Profit 20 20 20 40 40 40

Calculations

Calculation Process

Accounts Jan Feb Mar Qtr1

Sales 124.71 119.43 161.93

COGS 42.37 38.77 47.28

Margin 82.34 80.66 114.65 277.65

128.42

406.07

Dense Calculation

Data load from table

XX

XX

XX

###

###

###

XXXX XXXXXXXAfter calc of

Accounts

dimension

Sales COGS Margin Profit

Jan

Feb

Mar

Qtr1

YearAfter calc of Time

dimension

Sparse Calculation

Vermont -> Cola New York -> Cola

East -> Cola Calculated

blocks

Upper-level

blocks

Input blocks

Level zero

blocks

Calculation Order: All Dimensions

• First, Accounts

• Second, Time

• Third, remaining dense dimensions

• Fourth, remaining sparse dimensions

• Two Pass Calculation

Calculations

• Default Calc - simplest method

• Calc scripts

• Dynamic calculations

• Intelligent calculation

Commit Blocks

• Using Uncommitted Access

– When Commit Level is reached, blocks write to hard drive

• Default is 3000 blocks

• Setting Commit Blocks to Zero

– Writes at completion of the entire transaction

– Will dramatically improve calculation time

– Will fragment your PAG file during a calculation

Retrievals

Index Cache

Memory

Index pages

in

index cache

Disk

Index pages on

physical disk

New requests

Old requests

Data Cache

Disk blocks in

data cache

Disk

Disk blocks on

physical disk

Memory

New requests

Old requests

Internal Workings of Essbase: Aggregate Storage

BSO Limitations

• Financial applications are more densely populated so BSO works great in

those instances

• BSO engine can handle sparse data but on a ―limited‖ scale

• Outline size limited

• Batch times required for loads and calcs

Aggregate Storage Option

• Remember all the concepts we just learned:

– Dense / Sparse

– Index / page files

– Cache settings

– Block storageX

Aggregate Storage Databases

• Similar to a BSO database – outline, dimensions, hierarchies… BUT

• Different method for storing/calculating databases

• New storage kernel built to handle ASO databases

• Calculates 10-100x faster

• Stores up to 252 dimension combinations

Aggregate Storage Databases

• ASO addresses the following types of databases:

– Read-only*

• Write back to level zero available in 9.3.1

– ―Rack and stack‖

– Large dimensions

• New Types of databases are possible

– Customer analysis—data is analyzed from any dimension and there are potentially

millions of customers

– Procurement analysis—many products are tracked across many vendors

– Logistics analysis—near real-time updates of product shipments

– Market Basket analysis—products purchased along with other products

When to use ASO

• Database is sparse and has many dimensions, and/or the dimensions have

many levels of members

• Database is used primarily for read-only purposes, with few or no data

updates

• Calculation of the database is frequent, is based mainly on summarizing of the

data, and does not rely on calculation scripts

• Starting in Essbase 11x, ASO should be your default/starting idea

BSO vs. ASO

BSO Outline ASO Outline

End User Perspective

• End users won‘t care whether their database is ASO or BSO

• The way users access ASO is the same as BSO

– Excel Add-in

– Financial Reporting, Web Analysis, etc.

– Smart View Add-in

• Repeat… no differences (just more data/dimensions)

ASO Benefits from IT Perspective

• Faster load and calc times provide

– Lower hardware costs

– Lower maintenance costs

– Higher availability

• Small disk footprints

• Efficient tuning for storage and query response

How does ASO work?

• Simple question with not so simple answer

• Asked greatest minds in the business how ASO works and got the same

resounding answer:

• ―It‘s a black box‖ or ―it's top secret and hard to understand‖

• There must be a better answer!

ASO is Designed For…

• More dimensions and members

• Less time required for batches

– Fast aggregation of sparse data sets

– Faster loads

– Incremental loads

• Reduction in database footprint

Key Concepts

• Storage

• Sparse data

• Indexing

• Aggregation

• Nodes

ASO Storage (ROLAP in disguise)

• ASO can also be said to be ―ROLAP with a super fancy index

scheme that rules.‖

• Big difference between ASO and BSO and ROLAP is the ASO

storage mechanism

• ROLAP stores data in a table and indexes a combination key

between the rows

• Essbase stores the concept of a cube of data in multiple

dimensions or rather multiple keys

ASO Storage (ROLAP in disguise)

• It‘s a multidimensional index

• ASO takes it a step further and indexes the indexes in a way for more rapid

aggregation of data

• Storage is no longer in "blocks" but in highly optimized aggregation nodes

• Visualize it as an asymmetric fractal Christmas tree flattened out and then

indexed again

How does ASO work

• Load Data only at level 0

• Create aggregate views

• Algorithm selects and stores ―most taxing‖ queries

• Dynamic queries at runtime and increased

speed by leveraging nearest stored view

ASO Concepts

• Concept of stored and dynamic hierarchies

– Stored hierarchies can only aggregate

– Dynamic hierarchies can utilize formulas and advanced unary operators

• Formulas on dimension members use MDX syntax

• Pre-aggregated views can be defined to help query performance

• Aggregated design wizard helps you create aggregation scripts

• Outline paging helps you page portions of the outline in and out

of memory to assist in performance

• You can convert BSO outlines to ASO outlines using a wizard

Storage and Compression

Directory Structure

• Directory structure differs in both content and purpose from BSO

• Tablespaces are utilized to store data and metadata

– Default – stores numeric data (.dat file)

– Log – records database activity

– Metadata – stores metadata information about the objects in the database

– Temp – temporary working space for the Essbase kernel

Tablespace Overview

• Defines the database storage in the form of file locations

• Each ASO application has 4 tablespaces

– Default – database values

– Temp – temporary work space

– Log – transaction log files

– Metadata – database data structure

• File location specifies a physical disk space for storing database

files

• Each tablespace may contain one or more file locations

– Can span multiple physical drives and/or logical volumes

Manage Tablespaces

Sizing Tablespaces

• During the data load and aggregation process, data is stored in both the

Temp and Default directories

• ASO will always build the full .dat file in the temp tablespace while the default

tablespace still has the production .dat

• Hence, for your maximum drive size you have to plan on AT LEAST 3x your

max bloated .dat size if you want to be "safe" (buffer to temp to default)

Compression Dimension ASO

• In old releases, the Accounts dimension enabled database

compression

• Beginning in 9.3, the Accounts dimension is the default

compression dimension BUT you can choose a different

compression dimension

• Essbase helps you choose a compression dimension by

estimating what the database size would be depending on

which dimension is tagged as compression

• Time is an excellent candidate for compression dimension,

especially if you have fiscal year as a separate dimension

Calculations

Aggregating ASO Data

• For ASO databases, after data values are loaded into the level 0 cells of an

outline, the database requires no separate calculation step

• From any point in the database, users can retrieve and view values that are

aggregated for only the current retrieval

• ASO databases are smaller than block storage databases, enabling quick

retrieval of data values

• For even faster retrieval, you can precalculate data values and store the

precalculated results in aggregations

Aggregations – The Down Side

• Lengthy process

• Requires extra disk space

– Sometimes yes, but it is rare that default aggs are more than 40 percent the size

of the input data.

• You want to balance query time and storage space

Intelligent Aggregations for ASO

• You can define hard restrictions for a dimension

– Default (no restriction for primary hierarchy, no aggregation for alternate

hierarchies)

– Consider all levels

– Do not aggregate

– Consider top level only

• (you only query top level)

– Never aggregate to intermediate levels

• (you only query level zero or top dimension)

• Level based weighting – provide levels to consider

• Process

– ASO considers hints when creating aggregations

– Attempts to create the most useful aggregations based on

hints

Query Hints

• You can define ―soft restrictions‖ as a query hint

• Just select a representative member (any member)

• Essbase will take this into consideration when creating aggregation views

Query Hints

Design Considerations

• BSO

– Complex calculations and allocations

– Write back at upper levels from end users are required

• ASO

– Large analysis applications with many dimensions and members

– Rolling up and analyzing large volumes of data

• Both ASO and BSO

– Take advantage of the strengths of both database types

Consider Using Both ASO and BSO

Budget

Product SKU

Analysis

Part

itio

n

BSO

ASO

Consider Using Both ASO and BSO –

version 11

Budget

Product SKU

Analysis

Part

itio

n

BSO

ASO

ASO vs. BSO Recap

• What are the similarities between ASO and BSO?

– Building dimensions

– Loading data (level zero only for ASO)

– Retrieving data

• What are the differences?

– Many dimensions, many members

– No calc scripts

– Use MDX member formulas

– Aggregations for improved query performance

• Lots of improvements in System 9 and version 11

– Understand the limitations in early versions of Essbase ASO

– Don‘t miss the new features in 9.3.1 and 11x

Thank You!!

Edward Roske

[email protected]

BLOG: LookSmarter.blogspot.com

WEBSITE: www.interrel.com

TWITTER: ERoske