Navigating a Global R12 Upgrade

7/28/2019 Navigating a Global R12 Upgrade

1/44

Management & Technology Consulting

Management & Technology Consulting 11/4/2011 1


2/44

11/4/2011Management & Technology Consulting 2


3/44

Objectives:

Like for Like Technical Upgrade of Oracle Applications

11.5.10.2 12.1.3

Retrofit existing reports & customizations for compatibility and

usability on R12 Provide training and coordinate testing across 4 major regions

North America, Asia, India and Europe



4/44

Scope:

26 Ledgers

GL, FA, AP, AR, TAX, IE, CE, PO, INV, OM

40,000 User Base

955 Custom Objects Required Analysis 3TB Database

Global training and testing

7 Months



5/44

11/4/2011Management & Technology Consulting

5

Steering Committee: 15

2Leads

(Training &Testing)

12Developers

9FunctionalAnalysts

Global Business SMES: 50

2-3DBAs

Project Manager: 2 Technical Lead: 2


6/44

Decide upgrade approach and scope

Determine timeline & resources

Maximize Testing

Robust Cutover Planning

Testers should have a strong knowledge of business

processes

Issue Strategy

Communication



7/44

Training Approach

Onsite Classroom Training

Provided by Oracle University

Functional Analysts and Business Core TeamPhase 1

Train the Trainer

Provided by Jibe Consulting

Delta training of mandatory R12 functionalityPhase 2

End User Training

Provided by Amazon Business Core Team

Brown Bag Sessions, Onsite Training, Web ConferencesPhase 3



8/4411/4/2011Management & Technology Consulting 8

Training delivered prior to test cycles to ensure

maximum benefit

End user training delivered prior to UAT/Go-Live On-Site training was the most successful


9/44

Turnover within the organization led to training gaps

Follow up and obtain training plans from all areas

Small Changes = Big Questions Provide some level of training for all new/changed functionality

Face Time Business users who were offered onsite training/ web sessions were

the most successful in navigating through the new functions

Video Recordings

Timing Deliver training to coordinate with testing and Go-Live



10/44

TC1

Standard Oracle Functionality

Core Business Team

TC2

Introduced interfaces and customizations

Business SMEs

UAT

All business processes, custom objects andreports

End Users


3 Test Cycles


11/44

Challenges

Educating testers on how and what to test

Differences between unit testing and end to end testing

Volume of testing

How to measure progress

Solution

Conducted testing planning sessions with entire team

Held planning work sessions with individual teams

Assigned a finance owner to each end to end scenario



12/44

Global Reference Scenario Name Category Week 1 Week 2 Week 3

Target Test

Completion

Date Responsible

AR-US-001 Autoinvoice Import Program Vendor Returns 1 7/12/2011 Kala Jones

AR-US-002 Payables Open Interface Import Vendor Returns 1 7/12/2011 Kala Jones

AR-US-003 Prepayment Matching Program (Prepayments

Matching Program)

Vendor Returns1

7/12/2011 Kala Jones

AR-US-004 AMZ Load Vendor Return Data From FATE Vendor Returns 1 7/12/2011 Kala Jones

AR-US-005 AMZ Customers Interface load from Vendor

Returns

Vendor Returns1


AR-US-006 Customer Interface Vendor Returns 1 7/12/2011 Kala Jones

AR-US-007 AMZ Load AR Vendor Returns Invoices Vendor Returns 1 7/12/2011 Kala Jones

AR-US-008 AutoInvoice Master Program Vendor Returns 1 7/12/2011 Kala Jones

AR-US-009 View/Change Payment Terms Vendor Returns 1 7/12/2011 Kala Jones

AR-US-010 AMZ Vendor Return Create Reclass Journal Vendor Returns 1 7/12/2011 Kala Jones

AR-US-011 AMZ Vendor Returns Current State Summary

Report

Vendor Returns1


AR-US-012 AMZ Vendor Returns Current State Detail

Report

Vendor Returns

1


AR-US-013 AMZ AR Aging Report - Customer/Product

Summary

AR Unit Test1

7/8/2011 Andrew Nienhaus

AR-US-014 AMZ AR Invoice Aging Detail Report AR Unit Test 1 7/8/2011 Andrew Nienhaus



13/44


Blocks of time scheduled

by category

Color coding helped

testers identify when they

were needed


14/44

Module

Total ScriptsTotal Pass Total Fail Total Incomplete Attempted

# % # % # % # %

Accounts Payable 2556 2424 95% 132 5% 0 0% 2556 100%

Accounts Receivable 651 604 93% 48 7% 0 0% 651 100%

Cash Management 90 86 96% 1 1% 3 3% 87 97%

Fixed Assets 240 194 81% 0 0% 46 19% 194 81%

General Ledger 517 361 70% 86 17% 68 13% 449 87%

Inventory 86 86 100% 0 0% 0 0% 86 100%

Order Management 125 125 100% 0 0% 0 0% 125 100%Purchasing 380 375 99% 5 1% 0 0% 380 100%

eBusiness Tax 26 8 31% 18 69% 0 0% 26 100%

TOTAL 4671 4263 91% 290 6% 117 3% 4553 97%


Metrics were summarized for reporting

Drilldown by process and country


15/44

3 Week Test Cycle

Objectives

Test standard Oracle functionality

Business teams identified test scenarios & created test

scripts

Identify End to End Scenarios

Planning of future test cycles



16/44


Challenges SolutionManaging ExpectationsUsers not aware of what wasa customization

Functional Analysts present in the testingrooms Knowledge Transfer

Unexpected Issues withStandard FunctionalityCustom FoldersCustom ApplicationASLs no longer supported

Deleted all custom foldersCreated all new responsibilitiesConverted all ASL items to requisitiontemplates

Issues Logging ProcessMultiple entries for the sameissueEntries which did not clearlystate the problem

Process OwnerAssisted with reporting metrics and loggingissues

Test Cycle 1 - Challenges


17/44

5 Weeks 3 Weeks Unit/Transaction Testing

2 Weeks Integrated/End to End Testing

2343 Unit Test Scripts Executed

59 End to End Scenarios



18/44

Objectives Identify and resolve system issues

Identify training gaps

Complete generation/rework of test scripts

Exit Criteria

All Blocker issues resolved

All Critical issues identified and resolution ETA

determined



19/44


Challenges SolutionGlobal CoordinationTesters outside of the USstruggled with lack ofsupportExpectations were not clear

Bi-Weekly testing status callsWeekly calls with Functional AnalystsDaily communication of blocking issuesand progress

Testing InterfacesInitial testing was notsuccessfulFrustration

All interfaces thoroughly tested by FAprior to user testingInterfaces scheduled to ensure datapreparation was completeInterfaces tested globally


20/44

3 week test cycle Unit testing and Integrated testing

End Users responsible for testing

Global participation

4636 Test Scripts

Majority were in AP and AR

Performance testing



21/44

Objectives Resolve any remaining issues

Ensure business and system readiness for Go-Live

Complete performance testing

Exit Criteria

All Blocker Issues Resolved

All Critical Issues Identified

Needed to be resolved by month-end



22/44

Performance Issues Reports did not complete due to largevolume of data


Challenges SolutionFunctionality IssuesNew testers found newissuesCustomizations and reportstested for the first time

Additional resources addedEncouraged as much testing aspossible

Performance IssuesReports did not completedue to large volume of data

Identified key reports and didperformance tuningWorked with Oracle for solutions

Performance TestingFew resources and limitedtime to complete

Attempted load testing during UATRan several time consumingprocesses and monitored execution


23/44

Performance Issues Test, test, test and test some more

Tax and AP Data Issues

Multiple data fixes required

Test both new and upgraded data

PO/AP Reconciliation

Significant changes that were not tested by all geographies

Create Accounting Completely new concept, ensure users understand the

changes and implications to the month end process



24/44


Agenda Customization Planning Approach

Development Challenges

Performance Issues

Concurrent Manager

R12 Patch Monitoring and Tuning


25/44

Retrofit 900+ customizations in time for TC2 testing.Development time

P1 customizations ~2 months.

P2 customizations ~3.5 months

Minimize business downtime due to R12 patch

upgrade



26/44

Inventory Inventory all the customizations

Identify What can retire

What will upgrade cleanly

What needs to be re-implement in R12. E.g. check printing

Resource planning for the development team Finalize the list of customizations that will need to be retrofitted. Data model changes

tool used Document id 1290886.1. Cross reference with the customization list

Assign level of effort to broken customizations

Level of effort used: Low(8), medium(16) and high (40 hrs.)

Load resources appropriately

Assign resources to business modules. E.g. designated developer for bank integration

Provide technical training to the developers

Begin Development phase



27/44



28/44

Help understand the data model changes

Download document id 1290886.1


Data model comparison report


29/44

Check Print development Start early. Long development cycle Long test cycle with the banks Print issues

Report output shifted across printers even though they were the same make andmodel

Issue was due to the print drivers not being up to date for the printer. Some of theprinters were using altogether different drivers as a result of changes over time.

Used pdftops to stabilize the shifting of the output

Custom Application (Amazon.com) associated to CustomResponsibilities E.g. AMZ Purchasing Manager associated to Amazon.com instead of seeded

Purchasing application Caused breakage of self service forms in Payments, iProcurement and

iExpense LOV errors in certain standard reports E.g. Create Accounting Solution: Re-do responsibilities and remove amazon.com association. ~1200

responsibilities impacted Consideration for R12 upgrade



30/44

BI Publisher reports are memory hungry compared to

traditional Reports 6i Several month end reports failed to complete. E.g. Payables

Posted Invoice/Payments Register 45,000 pages of pdf output for one month

Started as 16 GB of report output.

Solution:

Oracle support provided a stripped down version of the XML

Output

Pre-allocated 2GB memory on the concurrent program definition.

Set Options = -ms2048m -mx2048m

3 properties need to be set to True for the report template under

XML Administrator Responsibility



31/44

Auto-Select and Build Payments programs erroring out

after hitting batches of 20K+ invoices

Solution:

Profile AP: Gather PPR Table Stats set to Yes at site level

Performance patch released by Oracle support

Create Accounting process running longer 20+ hrs.

impacting month close activities Solution:

Bug raised with Oracle support and patch received

Created a daily run schedule for Create Accounting job for each

geography

Assigned multiple processors to Create Accounting process to

decrease run time. Caution: optimize the number of processor

based on server and concurrent manager load. Screenshot on next

page



32/44

Recommend doing a load test during UAT testing

to uncover performance issues


Performance Issues (cont.)


33/44

Amazon uses 30+ custom managers to process jobsunder Concurrent Manager Used to segregate quick and long running jobs. E.g. Print POs and

Alerts(quick jobs) vs. Invoice import (long running) Separate managers for high volume jobs. E.g. Print AR Invoices.

Daily load of > 10K jobs

Pain Points Managers were constantly backed up. Several hundred jobs

sitting in pending state everyday 2 hrs. average wait time before jobs gets picked for processing Global users being impacted by spill over from other geographies Impacted several time sensitive activities E.g. month end reports,

PO printing SLAs Created negative user experience. Users constantly requesting IT

to bump up the priority of jobs



34/44

Observations: All workers not being utilized. E.g. between 10-18 workers would be

used instead of the full 20. Would use the full allotted only at times.

Solution: Reduced the number of concurrent managers from 30 to 18 Updated cache size of the managers to twice the number of workers Decreased sleep seconds of quick running jobs to 20 Increased sleep seconds to 120 for long running jobs

Summary Key is to minimize the number of managers

Optimize: Number of workers/processes

Sleep seconds

Cache size. Thumb rule is twice the number of workers



35/44

5 Days Downtime

Upgrade 3TB of database



36/44


Major Cutover Activities


37/44


Useful tool to communicate with the IT and Businessmanagement

Helps understand the long activities and focus on

reducing the time of those activities

Helps the team not to get confused with >5,000 linesof detailed cutover plan. Was used in conjunction

with the detailed cutover plan

Contingency built into major activities. Relieves

pressure if any activity gets delayed


38/44


Biggest contributors to the 5 days of downtime

R12.1.1 patch = 55 hrs.

Oracle data fixes for tax, AR invoices and AP =15 hrs. 10 data fixes totaling up to 15 hrs. of downtime

Long running due to the number of transactions being updated. E.g. 86

million rows in ap_invoices_all Missing Establishment_id in ZX tables

Ensure Oracle provides a script that can be run with parallel workers

Post-Upgrade reports = 8 hrs.

Issue was the runtime of the reports. Month end reports ran for >5 hrs


39/44


Technical upgrade is the process of upgrading 11i

environment to R12.1.3 Involves upgrading database to 11gR2 (done as a precursor

project) Laying technical stack for R12 Running R12.1.1 patch

Running R12.1.3 patch Post upgrade patches including one-offs identified by project

team

Recommendation: Plan on running at least 5 techupgrades Practice and practice more to make it a repeatable process Identify long running jobs and tune them


40/44

Business shut down activities Process all transactions

Submit all accounting transactions

Verify all interfaces are clean

Must be coordinated across all business teams

Smoke Testing

Key Interfaces and processes

Provide assurance to business and management that

system was ready

Ramp Up Slowly



41/44


Erratic runtime of AP/AR jobs. Oracle database

optimizer would change the sql execution plan andcause jobs to run longer This was due to:

Oracle dynamic sampling

Lack of database statistics

Use of temporary tables as part of upgrade

Solution: Pin SQL queries to a specific plan by using SQL profiles

Procedure:

Monitor long running queries in OEM Implement a better explain plan if recommended by Oracle OEM SQL profile the plan Export and use for next run


42/44


Technical Upgrade Process flow and tuning exercise


43/44



44/44

Oracle database name can only be in A-Z or 0-9

character format Rapidwiz failed to run during R12 prod upgrade. Did not like _

underscore in FIN_P db name

Solution: Changed the name to FINP. Caused the tnsnames file

to be changed for external systems.

Diligently follow R12 upgrade guide (Part No.

E16342-01) Excellent suggestions in the technical and functional areas

Good tips in pre-upgrade functional steps that help reduce

downtime.

E.g. Submit Accounting missed in 11i and caused one of job to run

for 7 hrs. instead of 3 hrs. during the tech upgrade

Navigating a Global R12 Upgrade

Documents