E-Business Suite Big Data Purge An Approach to Archive …ncoaug.communities.oaug.org/multisites/ncoaug/media/Events/NCOAUG...An Approach to Archive and Purge Financial Accounting

E-Business Suite Big Data Purge

An Approach to Archive and Purge

Financial Accounting Hub Data

Introductions

George Somogyi

IT Director

Societe Generale (Newedge)

Rey Mendez

Director, Oracle Applications

Avout Corporation

• Avout was formed in 2010 to

change the way consulting

companies delivered services

• Providing the Right Resource, at

the Right Time, at the Right Price

• Our Clients matter, our

Consultants matter, our

Executives matter, and the People

in our immediate and extended

community matter

• A world-leading multi-asset

derivatives broker

• 100% owned by Societe Generale

• Listed and OTC Clearing

• Agency Execution

• Prime Brokerage, incl. cross-

margining, financing, capital

introduction, research

• Access to 85 markets

2

Oracle Project Background

• Global Deployment Timeline

– North America – April 2011

– Canada – June 2011

– UK – November 2011

– France – January 2012

• Software Application Landscape

– Oracle e-Business Suite – R12.1.2

• Oracle Financial Accounting Hub, Oracle General Ledger, Oracle Payables,

Oracle Receivables, Oracle Assets and E-business Tax, Oracle Advanced

Global Intercompany Systems

– Hyperion HFM / FDM 11.1.2.3

– Reporting Toolset

• Essbase 11.1.2.3

• OBIEE 11.1.1.7

3

Fixed Assets (FA)

Advanced Global

Intercompany

System (AGIS)

TCA Payables

(AP)

Accounts

Receivable

Financial Application Architecture

Enterprise Reporting – OBIEE (Answers, Dashboards etc.)

NCR

Inte

gra

ted

Data

La

yer

Finance Data

Mart (FDM)

HFM

Essbase

Client Referential

Account Balances

and Shareholder

References

Account

Balances

COA Segments

Journal Header & Lines

Balances

General Ledger

Local GAAP, IFRS Ledger Options

Intercompany Detailed Journals

Financial Accounting Hub (FAH)

Subledger Accounting

Staging

Tables Pre-processing

Rules Engine

Net Business

Income

F10s

Futures

Balance Sheet

Trial Balance

Newedge Integrated

Data Layer

F10 Files

Ledger Options

Detailed Journals

Local GAAP, IFRS

Intercompany

Supporting Ref. w/balanceUser Tran. Identifiers

Rules Engine

Repository

Financial Accounting Hub / Subledger

Accounting

Payables

(AP)

Fixed Assets

(FA)TCA

Suppliers

Invoices,

Payments

Retirements

Depreciations

Transfers

Trading

Journals

Summary Journals

(IFRS, Local GAAPs)

Local CurrenciesDrill Back

General Ledger

COA Segments

Journal Header & Lines

Balances

EBTax

Advanced Global

Intercompany System

(AGIS)

Intercompany

TransactionsIntercompany

Payables, Receivables Journals

Finance Data

Mart

(FDM)

F10 information,

Oracle Code

Combinations

HFM

Account Balances

And

Supporting ref.

Balances

Enterprise Reporting - OBIEE (Answers, Dashboards etc.)

Pass Throughs

NCR

Suppliers

Procurement

Systems

Pass Through Trial

Balances

Oracle EBS Modules

Trading Source Systems /

Other Newedge Systems

Consolidation and Reporting Systems

Standard Product Interfaces

Custom Interfaces

Other Integration Points

Key

Asset

Systems

Legacy Trading Systems

Web ADI

Excel Uploads

Legacy

Lookups /

Referential

Manual Journal

Excel Upload

Essbase

Account Balances and Supporting

Balances

Manual Adjustment to only One Ledger

1

21

53

41

Concur (T&E)

ADP

Client Referential

Pre-processingStaging

Tables

Supporting Ref. w/o balance

PO Matching

T&E Invoice (approved

and paid)

Payment Information

Assets

(Conversion)

eUbix

Invoices

InvoicesTax

Engine

Net Business

Income

F10s

Futures

Balance Sheet

Trial Balance

Newedge Integrated

Data Layer

F10 Files

Ledger Options

Detailed Journals

Local GAAP, IFRS

Intercompany

Supporting Ref. w/balanceUser Tran. Identifiers

Rules Engine

Repository

Financial Accounting Hub / Subledger

Accounting

Payables

(AP)

Fixed Assets

(FA)TCA

Suppliers

Invoices,

Payments

Retirements

Depreciations

Transfers

Trading

Journals

Summary Journals

(IFRS, Local GAAPs)

Local CurrenciesDrill Back

General Ledger

COA Segments

Journal Header & Lines

Balances

EBTax

Advanced Global

Intercompany System

(AGIS)

Intercompany

TransactionsIntercompany

Payables, Receivables Journals

Finance Data

Mart

(FDM)

F10 information,

Oracle Code

Combinations

HFM

Account Balances

And

Supporting ref.

Balances

Enterprise Reporting - OBIEE (Answers, Dashboards etc.)

Pass Throughs

NCR

Suppliers

Procurement

Systems

Pass Through Trial

Balances

Oracle EBS Modules

Trading Source Systems /

Other Newedge Systems

Consolidation and Reporting Systems

Standard Product Interfaces

Custom Interfaces

Other Integration Points

Key

Asset

Systems

Legacy Trading Systems

Web ADI

Excel Uploads

Legacy

Lookups /

Referential

Manual Journal

Excel Upload

Essbase

Account Balances and Supporting

Balances

Manual Adjustment to only One Ledger

1

21

53

41

Concur (T&E)

ADP

Client Referential

Pre-processingStaging

Tables

Supporting Ref. w/o balance

PO Matching

T&E Invoice (approved

and paid)

Payment Information

Assets

(Conversion)

eUbix

Invoices

InvoicesTax

Engine

Net Business

Income

Futures

Balance Sheet

Trial Balance

Legacy Trading Systems

Pass Throughs

F10s

Web ADI

Excel Manual

Journal Uploads

4

EBS Business Landscape

Ledgers • North America

• US GAAP, primary ledger, ledger 2024

• US IFRS, secondary ledger, ledger 2025

• Canada, primary ledger, ledger 2028

• UK: IFRS primary ledger

• French

• FR GAAP, primary ledger, ledger 2053

• FR IFRS, secondary ledger, ledger 2091

Reporting Components

• One Global Chart of Accounts

• Reduce FX exposure globally with the use of a single set of exchange

rates (over 280 combinations)

• Global regulatory reporting requirements

5

Daily Processing Volumes

• Core Oracle application is the Financial Accounting Hub (FAH)

• The event business model in FAH is used to transform front office trading

system transactions to accounting events

• On the average, we receive 1,500,000 summarized trade transactions daily

from our front office systems

– A single trade transaction is transformed into anywhere from 2-8

accounting transactions

– On a given day, in excess of 20 million records are generated in FAH

across the following tables:

• XLA_TRANSACTION_ENTITIES

• XLA_EVENTS

• XLA_AE_HEADERS

• XLA_AE_LINES

• XLA_DISTRIBUTION_LINKS

• XLA_AE_SEGMENT_VALUES

• GL_IMPORT_REFERENCES

• XLA_AC_BALANCES

• XLA_AE_HEADER_ACS

• XLA_AE_LINE_ACS

6

EBS Technical Performance Metrics

• All trading activity from the prior business day must be processed and

reported on the next business day

• In the U.S. , we are under regulatory requirements to generate

financial statements on a daily basis (i.e., daily closings)

• Under our service level agreements with accounting, all trading activity

must be processed by 5:00 a m. (Chicago time)

• As the earliest we receive files from the trading systems is 2:00 a.m.,

all processing must occur within a 3 hour window between 2:00 and

5:00 a.m.

• To meet these tight SLAs, we choose Oracle Managed Cloud Services

to host our Oracle EBS Applications which run on an Exadata X2-2

7

EBS Technical Architecture

• Hosted by Oracle Managed Cloud Services

• Production Architecture

– Software Platform

• Oracle EBS version 12.1.2

• Database version 11.2.0.4.0 - 64 Bit

• OS – Linux

– Primary Production Environment

• Exadata X2-2

• 2 DB Compute Notes / 7 Storage Cells

• 2 application tiers

– Production DR

– Four non-production environments

8

Understanding our Data Volume

• We anticipated long overnight processing cycles

• In preparation for our first deployment in North America, we

conducted extensive performance tests over four months to right

size our hardware platform

• Before the North America go-live, we anticipated our data growth

to be accelerated and storage requirements to peak in four years.

• To plan for the future, we conducted an assessment of Oracle

EBS archive software vendors with help of an independent

consultant

9

Newedge Storage Projections

• We projected 119 TB of storage used by January 2014.

• This was under the assumption that un-summarized trade

transaction would be brought into Oracle

• Only 18 TB were budgeted

10

Database Table Distribution

XLA_AE_LINE_ACS 12%

XLA_DISTRIBUTION_LINKS 19%

XLA_AE_HEADER_ACS 0%

XLA_AE_LINES 17%

XLA_AE_HEADERS 13%

GL_IMPORT_REFERENCES 10%

XXNE_F10_GL2_COMBINATION 4%

XLA_EVENTS 3%

XLA_TRANSACTION_ENTITIES 2%

Remaining Tables 16%

Database Table Distribution

85% of database size represented data from FAH / XLA tables

11

Impact of our Data Volume

• After three years and when all regions were live, we started to

experience longer batch processing cycle times and a

degradation of performance for reporting

• General system stress and occurrence of software bugs

• Storage costs continued to increase

• The size of our production database grew from 2 to 8 TB. The

data in the FAH application represented 85% of our storage

12

FAH Volume Analysis (2011 – 2014)

Summary by Region

Region Number Records %

United States 14,100,000,000 94%

United Kingom 450,000,000 3%

France 450,000,000 3%

15,000,000,000 100%

13

Data Reduction Roadmap

• Objective

– To reduce storage costs

– To improve system performance

– To provide a foundation for future state archive / lifecycle

management activities

• Major Activities

– Internal projects

– Global data retention requirements gathering

– FAH Archive and Purge

– Database Reorganization

14

Data Reduction Roadmap

– Internal Projects

• Aggregated inbound transactions from trading systems to lower

volume coming into EBS / FAH (versus detail trades)

• Purge FAH interface tables (GL_XLT). These tables consumed a

larger amount of storage than anticipated. New tables were created

with each nightly batch process and were not purged without explicit

maintenance

• Remove inbound and outbound interface files from test

environments

• Apply compression to custom reporting tables.

• Reduced foot print of non-production environment s by sub-setting

after instance was refreshed from production

15

Data Archive Requirements / Strategy

• We needed to understand how much data could be archived

• It was essential to engage the business to define their data retention

requirements

• We asked the business to identify different types of data using the

following data retention categories:

1) Real Time Access. The data is available immediately, directly

accessible from the production system

2) Near Real Time Access. The data is available immediately from an

archive database, but with slower performance and response time than

the production system

3) Offline Access. The data is available for recall from an offline data

storage. Within 48 hours

4) No Data Retention: Data does not need to be retained for this time

frame

16

Database Reorganization

• Data reorganization was necessary to realize the storage reduction from the

FAH Archive / Purge project

• Performed by OMCS

• Reorganization of entire production database

• Objects impacted were tables, segments, and indexes

• The largest schema reorganized were “Apps” and “Bolinf” (custom schema)

• We required reorganization to be completed in under 24 hours

• During re-organization, primary database was disconnected from the stand-

by environment (DR)

• Extensive validation activities were performed

– All indexes were rebuilt and validated

– Reviewed for invalid objects

– Thorough testing to ensure no impact on production processes

• Stand by / DR was rebuilt

17

Actual Storage (2011 – 2014)

• By mid 2013, we realized storage was at an unsustainable level and

needed to be reduced

FAH Purge

Database

Reorganization

18

Storage Savings After Data Reduction

Initiative Production / DR

Storage Savings

Non-Prod Storage

Savings

Total Storage

Savings

1 - BAU Maintenance

0.5 1.0 1.5

2 – FAH Data Archive / Purge

6.4 3.6 10.0

3 - Database Reorganization 2.0 6.0 8.0

19.5

19

FAH Data Purge Project

• Revisited 2010 assessment of Oracle EBS archive software vendors

• Vendors Considered

– Software and professional services

• Solutions offered

– Full licensed model

– Subscription basis

– Build to suit: custom software owned by SG

• Cost Estimates

– Ranged from $145 – $180K

– Excludes second year and beyond subscription cost

• Chose professional services

– Lower cost

– Ownership of solution

– No on-going maintenance cost

20

Financial Accounting Hub

Archive and Purge

21

Financials / Archive Logical Architecture

Oracle Business Intelligence Reporting (OBIEE)

Integrated Data Layer / Union Views

Reporting Tables and

Materialized View

Oracle Financials R12

Financial

Accounting Hub

General Ledger

Modules (Fixed Assets, A/P,

and A/R)

Archive / Reporting Database

Archive - Financial

Accounting Hub

Future State:

Archive - General

Ledger

Future State:

Archive - Modules (Fixed Assets, A/P,

and A/R)

Reporting Tables and

Materialized View

22

Background

• Project start date: Nov 11, 2013

• The right resource, at the right time, at the right price

– FAH Architect, PM/Testing Lead, Developer/DBA

• Newedge brought in Avout to help implement a FAH archiving and

purging solution

– The solution will allow Newedge to significantly reduce the size

of the production environment

– The solution will adhere to Newedge’s global data retention

requirements

– The solution will be repeatable, in other words, can execute the

solution again for the following years

– The solution must be deployed in 36 hour window

– The project must minimize impact on users and Newedge

business

23

Benefits & Scope

• Project Benefits – Storage cost savings

– Important element to Newedge’s archiving and purging strategy

– Reporting performance improvement

– Capable of phasing the implementation by splitting archiving from

purging activities

– No downtime required for the archiving step (performed during off

hours)

• Scope

– Archive and Purge Oracle Financial Accounting Hub application data for

all accounting years 2012 and prior

– Provide solution that addresses all regional data requirements

24

Solution Requirements

• Business Requirements

– Meet regulatory filing requirements

– Business Policy Compliant

– Adhere to Newedge Security Policies

– Archive Reporting Solution

• IT Requirements

– Oracle Referential Integrity Compliant

– Data Integrity

– Configurable Solution

– Easily Transport Data

– Repeatable Process

– Contingency Plans

25

Challenges Considered

• Considered Straight Partition Exchange of Data

– Challenge is that SLA tables are partitioned by

Application ID and not accounting date

• Referential Integrity: Needed to ensure reversing

journals that have accounting events crossing over a

year-end would remain with the non-archive records

• Temp table space limitations. Because the solution

required minimum downtime, the archive step would take

place while the system was still operating

• 36 hour cut-over window

26

Solutions Considered

Drop Storage

• Pros

– End to End processing time is

shorter

• Cons

– Removes indexes and

privileges

– Greater risk that Oracle would

not support if issues found

– Moving data from backup

tables into newly created SLA

tables.

Truncate Storage

• Pros

– No movement of data required for

the 2013 year FAH data.

– Lower risk of support issues

– No issues with index rebuild and

privileges

• Cons

– End to End processing time is

longer

27

Solution Components

• Design calls for archiving to take place prior to purging. For validation

purposes we needed to make sure what was in “Archive” was not in “Base”

and vice versa

• Key business rule: Archive accounting dates had to match purge dates

• Archive Process

1. Count records in archive table and base table

2. Create 2 driver tables based on the date ranges that the users would

like to archive (parallel processing speeds up the process). Create

indexes on driving tables

3. Insert data into 11 archive tables by selecting data directly from

related base table and limiting data based on the respective driving

table

4. After necessary indexes and analysis of archive tables, calculate final

counts in archive table and confirm no integrity violations found

28

Solution Components (continued)

Purge Process

1. Count records in the base table

2. Insert records into purge tables

i. If partitioned source table, select data from FAH partition where

data does not exist in the respective archive table and insert into

temp purge table

ii. If non-partitioned table, select data from base table where the

data does not exist in the respective archive table and insert into

respective purge table

3. Count records : purge + archive = base

i. If no match, print error DO NOT proceed

ii. If counts match, proceed to next steps

4. Truncate base table

i. If partitioned table, truncate FAH partitioned

ii. If non-partitioned, truncate full table

29

Solution Components (continued)

Purge Process - continued

5. Insert into base tables

i. If partitioned table, exchange partition from purge table into the base table

ii. If non-partitioned table, insert all date from the purge table into the base

table

6. Compare new base table counts = purge temp table counts

i. If counts are the same, success

ii. If counts are different, then error

7. Rebuild index partitions/ indexes

i. If partitioned table, rebuild index partition(s) on the table

ii. If non-partitioned table, rebuild indexes

30

Project Approach

• Created flexible schedule to work around business

deadlines and constraints

• Divide and Conquer: separated archive work-stream

from purge work-stream

• Daily status reviews of testing progress

– Involved all regional business representatives in

testing

• Formal signoff of testing results

• Clear communication at all levels

31

Testing

• Partnered with business to develop test plans, included regression

testing of key processes

• Involved global testing resources

• Clearly identified roles and responsibilities

• Divided testing between the archive /purge solutions

• Extensive Testing performed (Unit & UAT)

• Daily Status updates

– Reviewed results and discussed next steps

– Utilized green, yellow, red dashboard – immediately addressed

red issues

• Performance testing – Validated performance of key processes

32

Key Deployment Activities

1. Solution Blueprint into Action

i. Provided Operational Procedures Guide

• Sequential steps with expected timing of activities

ii. Configuration Guide

• Provided step by step configurations of the solution

2. Cutover Plan

i. Worked with team to give complete picture of activities and there

dates to be performed by Avout, Oracle, and Newedge with critical

path items understood

3. Contingency Plan

i. Defined a plan to rollback if we ran into any issues with the purge

process

4. Communications – Emails, Calls, Meetings

33

Lessons Learned

• Testing environment should be EXACTLY like production

– Did not properly account for impact of the DR environment

during production cut-over

• Prior to beginning the testing phase of the project, receive

commitment from business on resources

• Importance of advanced planning and communication with OMCS

34

Questions

• Questions?

• Contact Information:

– George L. Somogyi, Newedge

• E-mail: [email protected]

– Rey Mendez, Avout

• E-mail: [email protected]

35