Parallel Upgrade of PeopleSoft Applications and Oracle Database: Tips from MetLife Gopi Kotha, Software Systems Specialist, MetLife Asha V. Santosh, Lead.

Parallel Upgrade of PeopleSoft Applications and Oracle Database: Tips from MetLifeGopi Kotha, Software Systems Specialist, MetLife

Asha V. Santosh, Lead PeopleSoft DBA, MetLife

Navin Lobo, Technical Leader, Solution Support Engineering, Oracle

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 3

Program Agenda

Oracle Advanced Customer Services

Overview of MetLife

MetLife’s PeopleSoft Environment

Upgrade of PeopleSoft Applications and Oracle Database

Tips from MetLife

Q&A

1

2

3

4

5

6


Mission Critical Support Services for all Oracle Applicationsand Technologies

Service Highlights:• Oracle Advanced Monitoring and

Resolution: Remote monitoring and resolution across the entire IT stack

• Oracle Solution Support Center: Technical Account Manager and a team of advanced support engineers provide 24/7 personalized support

• Business Critical Service:A combination of 24x7 fault monitoring, priority handling of IT service requests, quarterly reviews and patch installation

Maximize Availability,Accelerate Return onInvestment andOptimize Performance

Tailored mission critical support services for customers with complex

IT requirements Strategic relationships focused on

collaborative, long term partnerships


Oracle Solution Support CenterCUSTOMER-FOCUSED SUPPORT TEAM

• 24X7 Dedicated toll-free number• Focused Team of Support Engineers• Led by named Technical Leads• Faster resolution building on customer affinity• Root Cause & Corrective Action Plans

TECHNICAL ACCOUNT MANAGER

• Customer Single Point of Contact for Support Issues• Escalation Management• Proactive Support Reviews

PRIORITY SERVICE REQUEST HANDLING

• Faster Service Request response times• Prioritization of Service Requests in Support work

queue• Escalations of Service Requests via Technical

Account Manager

PREVENTIVE SERVICES

• Preventive advice delivered based on specific Customer Knowledge

• Advanced Diagnostic Tools providing Actionable Recommendations

• Patching, Configuration & Product Use Guidance

BENEFIT

Increase Availability

Improve Performance & Reliability

Reduce Operational Cost

Reduce Risk


Oracle and MetLife

•Oracle Advanced Customer Support Services– Engaged at MetLife for over 4 years

– Solution Support Center service focusing on Oracle Database environments

– Assisted Services offering escalation management for PeopleSoft environments

•Introducing MetLife– Gopi Kotha, Software Systems Specialist ([email protected])

– Asha V. Santosh(, Lead PeopleSoft DBA ([email protected])


Overview of MetLifeWho are we?

• MetLife, Inc. (NYSE: MET) was founded in 1868• Leading global provider of insurance, annuities, employee benefit and asset

management programs• Serving 100 million customers in more than 50 countries• Leading market positions in US, Japan, Latin America, Asia, Europe and

Middle East• More than 65,000 employees world wide• 42nd on Fortune 500® list and 69th on Forbes Global 2000 SM ranking of

the world’s largest public companies, as of 2014


Market Leadership

1Based on life insurance in force according to A.M. Best (U.S.); Based on direct premium according to EstadisticAMIS (Mexico); Based on premiums according to Federal Board of Insurance Supervision (Russia); Based on combined direct premium according to Superintendecia de Valores y Seguros (Chile); Based on combined direct premium according to Superintendencia de Seguros de la Nacion (Argentina); 2Issue Date: 5/23/2011; 3LIMRA and MetLife Research; 4Nikkei (Japan); Financial Supervisory Service (Korea); SUSEP (Brazil); 5AXCO


MetLife’s Role in the World


• Functionality– Remove Book Code from CF– Implement functionality that:• Improves and automates business processes• Reduces customizations

• Enhance Processes – Review current business processes and eliminate

manual steps– Enhance system accuracy– Maintain systematic audit controls– Implement UNICODE database

• Efficiency – Implement new tools such as WorkCenters,

Application Search, Query pivot grids, and Form and Approval builder to reduce manual processes.

• Reduced TCO – Implement new tools like Chartfield request process,

configuration migration, Audit and Testing Framework

Value in PeopleSoft Financials Upgrade


MetLife's PeopleSoft Oracle FootprintInfrastructure Overview

• Oracle® RDBMS on AIX with EMC storage arrays; using Oracle Data Guard for DR/Standby database

• App & Web servers on AIX; NT for Process Scheduler to print crystal reports

• PeopleSoft Financials(modules GL/AP/PO/TE/EX/PC) and GL sub ledgers (CLASIC/STAR/RIBS)

• Reporting using a logically replicated environment using Shareplex software

• STREAMs to replicate setup/config tables to multiple targets from 1 source

• IBM’s Tivoli Workload Schedule Manager for batch trails


PeopleSoft Upgrade: Target Versions

H/S Component Old Version Target Version

AIX Hardware P6 P7

Oracle Database 10g 11.2.0.3

PeopleSoft Application 8.8 9.1

People Tools 8.49.10 8.52.07

Shareplex Software 5.3 6.5

Markview by Kofax 5.9 7.1

Vertex Tax Software 4.1 6.0


PeopleSoft Environment Volume Metrics• About 700 Business Units / Legal Entities For Americas region• Roughly 50,000 Journals processed in a month with more than 12 million

Journal Lines in Corporate Ledger • In one Sub Ledger we process 30,000 Journals with 120 Million Journal lines

every month• Corporate Ledger is a single instance with multiple modules for corporate

accounting• 10+ Integrations to other systems Internal/External• 6 hrs. batch window to process all the data every night


Before Cutover Weekend• 3 months plus of prep work on new AIX hardware, Oracle and PeopleSoft

software installs, conversion of 8.8 journals• PeopleSoft Journal conversion using two methods, convert data up to April

2013 before cutover (8.x), May & beyond during 3 day cutover (9.1)• Corporate Ledger & Reporting environment managed by SHAREPLEX

software• 3 Sub Ledgers sharing the same set of Chartfields and replicated from

corporate ledger using Oracle Streams• 2 Middleware databases feeding PeopleSoft environments• Imaging (Kofax Markview) & Vertex tax applications


Cutover Weekend and Volume• Timeline: Everything to happen in 72 hrs.(May 24th to 27th 2013 - 3 days) • Upgrade of 8 PeopleSoft and Non PeopleSoft applications and data

conversion to new CF– 4 TB of data load into 6 databases (preloaded historic 10 TB on to pre upgraded

11GR2 DBs)– Archive and purge all data before 2010 (Query option of Data pump)

– Conversion of Character set to UNICODE (used Data pump)– Data conversion - All the 8.8 journals got converted from old to new Chartfields– Reconciliation between 8.8 and 9.1 application, Ledger and Sub Ledgers – Build DR database for all (used SRDF for initial build) – Activate application servers (Tuxedo) and Web servers (Web logic)


During Cutover – Challenges Faced • Performance issues with data load and conversion for the month of May 2013

• Network Slowness – During Shareplex Replication instantiation from GL to reporting

• Process the Friday night trail (24th Trail) on Monday (27th) on the new system

• DR build – Moving 10TB across the data centers from primary to standby was a massive challenge

• RMAN backups slowed down due to volume (archive backup from primary DB server and database backup from BCV mirror copy)


Our Design approach and Tips• PeopleSoft /SQL Query Performance Tips• Statistics collection approach• Database Design Tips • Use Database Features


PeopleSoft / SQL Query Performance TipsStats Gathering

• Stats collection configuration:– Disable Automatic Maintenance Task Management–Manual Table Statistics Gathering– Concurrent Statistics Gathering – System Statistics Generation – Statistics at Runtime for Temporary Tables – Turning on %UpdateStats– Seed and lock Statistics for Temporary Tables

• COBOL or AE processes update table statistics using the FORCE option


PeopleSoft / SQL Query Performance TipsHistograms

• Candidate columns for Histograms within PeopleSoft Applications PS_JRNL_LN - Selectively enable histograms for certain tables• Since the typical PeopleSoft Application uses SQL with binds, we TURNED

OFF Histograms, to get an optimal execution plan• Let AE program use the meta-SQL “%UpdateStats” after rows are

populated to a temporary table. Doing so would update statistics in the temporary table, before the table is used in the SQL


Database Design TipsPartitioning

• Implemented extensive partitioning to enable performance enhancement and future periodic archiving & purging of historic data without impacting the performance of the applications in general (3-19 tables)• Multiprocessor with increased “db_writers” and a degree of parallelism,

eliminating the Buffer Busy Waits on inserts• Eliminate I/O bottlenecks and achieve high throughput by interval

partitioning (automatic range) GL tables, such as PS_JRNL_LN( journal_dt) and PS_LEDGER (fiscal year)• Partitioning pruning capabilities allow us to process only the relevant data


Database Design TipsPartitioning

• Change selective indexes from local to global– As a standard kept all partitioning indexes as local except certain indexes

(PSDJRNL_LN) as global where GLEDIT is doing data selectivity across partitions– It is best if the data that is being processed does not restrict to the same partition


Database Design TipsAdvanced Compression

• Used Advance Compression (OLTP option) at partition level for all transaction tables used for reporting• Space savings of up to 50%• Help reduce I/O for reads• Up to 30% faster selects with minimal impact to insert and delete

operation. Some impact to updates, so implement for historic partitions• Used Secure file compression for LOBs to control the growth of about

200GB/Week to under 500MB.


Use New Oracle Database Features with PeopleSoft• Partitioning and compression is not a customized option for PeopleSoft

applications anymore - makes maintenance easier• Interval partitioning along with incremental stats for ease of management • Automatic Memory and UNDO management• Use dynamic sampling with degree 4 to help with unpredictable volume

spikes.


Post Upgrade Challenges• Month End/Quarter End/Year End Close – performance issues• Volume and Stats management for partitioned tables – Periodically Seeded

stats for every new month partition to account for volume spikes with scale factor• Automate the rolling, archiving and purging process• Collect STALE stats for select GLs before nightly trails• Decommission of 56 servers to keep the TCO of Upgrade manageable


References• People books• Oracle documentation• MetLife testing and benchmarks

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Next StepsContact Oracle AdvancedCustomer Support Services

http://www.oracle.com/acs [email protected] Q&A


Parallel Upgrade of PeopleSoft Applications and Oracle Database: Tips from MetLife Gopi Kotha, Software Systems Specialist, MetLife Asha V. Santosh, Lead.

Documents

oracle slide

oracle andor

oracle applications

oracle advanced monitoring

oracle database environments

oracle database tips

metlife asha

metlife research