<Insert Picture Here>
Upgrading to E-Business Suite R12 – Best Practices
Eugene Weinstein Senior Director, EBS/Fusion Env. Operations
Sharon Yam Director, Applications Technical Services
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
Objectives
� Provide an overview of E-Business Suite R12
upgrade process
� Discuss the best practices of E-Business Suite R12
upgrade including downtime reduction techniques
(based on multiple test upgrades on customer
volume E-Business Suite databases and internal E-
Business Suite databases)
� Share experiences of upgrading internal Oracle
Global Single Instance to E-Business Suite R12
using best practices
<Insert Picture Here>
Agenda
• Overview of R12 Upgrade
– R12 File System
– Supported Upgrade Paths
– R12 Upgrade Flow
– Technology Improvements
– Upgrade by Request
• Best Practices
• GSI Upgrade Overview and Experiences
• Summary
• Q&A
Overview - R12 File System
iAS 10.1.3
ORACLE HOME
Developer 10.1.2
ORACLE HOME
Database
ORACLE HOME
RSF 10.1
Apache 1.3
OC4J
RSF 10.1
Forms 10
Reports 10
RSF 10.2
RDBMS
Components
APPL_TOP
COMMON TOP
INSTANCE TOP
MAJOR COMPONENTS
� Java Home: 10.1-based
iAS 10.1.3 – new
� C Home: 10.1-based
Developer 10 standalone install of
AS 10.1.2
phase2 – new
� Database Home: 10.2
� Appl Top: Applications’
code staging area
� Common Top: Runtime location for
Java, HTML
� Instance Top: configuration and
run-time generated files – new
Multiple instances can easily
share the middle-tier home
Overview - Supported Upgrade Paths
Notes:
1) Please see Metalink Note 403339.1 for supported combinations of Apps and RDBMS for R12 upgrade.
2) Release 11.5.10.2 includes systems installed with Rapid Install 11.5.10.2 and those upgraded by using the 11.5.10 CU2 maintenance pack.
PPPPAAAATTTTHHHH AAAA
--------SSSSiiiinnnngggglllleeee----PPPP
hhhhaaaassss
eeee UUUU
ppppggggrrrraaaa
ddddeeee
11.0
Upgrade to
11.5.10.2
Apps and DB Upgrades must be completed
during the same critical downtime.
11.5.1–11.5.6
Upgrade to 11.5.10.2
11.5.9cu211.5.10.2
9iR2, 10gR1
R12
10.2.0.3
PATH C PATH C PATH C PATH C ---- DualDualDualDual----Phase UpgradePhase UpgradePhase UpgradePhase Upgrade
Phases may happen in
separate critical downtimes
Phase 1: DB Upgrade
Phase 2: Apps
Upgrade
11.5.9(cu1)
11.5.10(cu1)
9iR2, 10gR1
11.5.711.5.8
9iR2
PPPPAAAATTTTHHHH BBBB –––– DDDDBBBB PPPPaaaattttcccchhhheeeessss ++++ AAAAppppppppssss
UUUUppppggggrrrraaaaddddeeee
DB one-off patches and
only APPS upgrade are
required11.5.9cu211.5.10.2
10.2.0.3
Overview – R12 Upgrade Flow
Start the upgrade
EBS atleast 11.5.7? Upgrade to 11.5.10.2
Prepare for EBS upgrade (Chapter 1
“Planning for an
Upgrade”)
Perform all applicable steps in Chapter 2 “Preparing for the Upgrade” except the last step of migrating the database
EBS at 11.5.9
CU2 or 11.5.10.2?Database at 10.2.0.3
Option to upgrade
database to 10.2.0.3
Perform Chapter 3 “Upgrading to Rel. 12”
steps up to migrating
the database step
Database at
10.2.0.3?
Database at
10.2.0.2?
Upgrade database to
10.2.0.3 using Rapid Install Oracle Home
Upgrade database
to 10.2.0.3
Apply Database
Patches
Complete
Chap. 3 steps
(including Run Upg. driver)
Complete
Chapter 4
“Post-upgrade Tasks” Yes
No No
Yes
Yes
No
Yes No
No
Yes
Upgrade
Complete
Overview – R12 Upgrade Flow (cont’d)
Start the upgrade
Perform all applicable steps in Chapter 2
“Preparing for the
Upgrade”
Prepare for EBS upgrade (Chapter 1
“Planning for an
Upgrade”)
Perform Chapter 3
“Upgrading to Rel. 12”
steps up to migrating the database step
Upgrade database to
10.2.0.3 using Rapid
Install Oracle HomeNote: 403339.1
Apply Database Patches
Complete Chapter 3 steps including “ run
upgrade driver” step
Complete Chapter 4 “Post-upgrade Tasks”
Upgrade Complete
R12 Upgrade Flow for an E-Business Suite instance
11.5.8 with 9.2.0.x database
Overview – R12 Upgrade Flow (cont’d)
Start the upgrade
Perform all applicable steps in Chapter 2 “Preparing for the
Upgrade”
Upgrade Complete
Prepare for EBS upgrade (Chapter 1
“Planning for an Upgrade”)
Perform Chapter 3 “Upgrading to Rel. 12”
steps up to migrating the
database step
Apply Database Patches during the migrate
database to Oracle 10g
Rel 2 step in Chapter 3
Complete Chapter 3
steps including “ run upgrade driver” step
Complete Chapter 4
“Post-upgrade Tasks”
R12 Upgrade Flow for an E-Business Suite Instance
11.5.10.2 with 10.2.0.3 database
Overview – R12 Upgrade Flow
Start the upgrade
Perform all applicable steps in Chapter 2
“Preparing for the
Upgrade”
Prepare for EBS upgrade (Chapter 1
“Planning for an
Upgrade”)
Perform Chapter 3
“Upgrading to Rel. 12” steps up to migrating the
database step
Upgrade database to
10.2.0.3 using Rapid Install Oracle Home
Refer note: 403339.1
Apply Database Patches
Complete Chapter 3
steps including “ run upgrade driver” step
Complete Chapter 4
“Post-upgrade Tasks”
Upgrade Complete
R12 Upgrade Flow for E-Business Suite Instance
11.5.10.2 with 10gR1 (Single Phase)
Overview – R12 Upgrade Flow (cont’d)
Start the upgrade
Perform all applicable steps in Chapter 2
“Preparing for the Upgrade”
Prepare for EBS
upgrade (Chapter 1 “Planning for an
Upgrade”)
Upgrade database to 10.2.0.3.
Refer note: 403339.1 for
details
Continue with the R12 Upgrade or
Use the 11i /10.2.0
System
Apply Database Patches during the migrate
database to Oracle 10g
Rel 2 step in Chapter 3
Complete Chapter 3
steps including “ run
upgrade driver” step
Complete Chapter 4 “Post-upgrade Tasks”
Upgrade Complete
R12 Upgrade Flow for an E-Business Suite
11.5.10.2 instance with 10gR1 database
(Dual Phase)
Overview – Technology ImprovementsApplications DBA (AD) Improvements
• AutoPatch replaces AutoUpgrade tool
• AD parallel infrastructure improvements for work
distribution of Large Table Update
– Divides large table into chunks
– Use separate processes in parallel to update each chunk of
data
• Inclusion of Gather Auto Stats job in R12 Upgrade
driver itself to keep statistics up-to-date after the
upgrade
Overview – Technology Improvements Applications DBA (AD) Improvements (continued)
• Check file and check file equivalence
• sqlplus_parallel directive to eliminate the contention
between jobs executing parallel query
Overview – Technology Improvements
Performance Improvements
• Optimizer dynamic sampling (for objects with no
statistics)
• Gather Auto option for Gather stats program to gather
only new or changed statistics using table monitoring
feature
• Converted non-critical jobs as concurrent manager
requests to reduce overall downtime
• R12 unified upgrade driver provides an option to defer
compilation of PL/SQL packages during creation
Overview – Upgrade by Request
• Option to additionally upgrade historical data that has not been upgraded during the initial upgrade process (which upgrades by default one fiscal year worth of data)
• Historical data can be upgraded anytime when system is up or down
• Following products historical data can be upgraded at a later date– Financials and Procurement
– Projects
– Supply Chain Management
– CRM (run scripts manually)
Overview – Upgrade by Request (cont’d)
• Upgrade of historical data depends on product. For some products only SLA data will be upgraded and for others both transactions and accounting data will be upgraded.
• Implementation is a two step process:1. Set range of periods of the historical data to be upgraded
before R12 Upgrade and run pre-upgrade concurrent program
2. Run SLA post upgrade (upgrade by request option) after R12 upgrade
• Review Appendix G in R12 upgrade manual for more details
<Insert Picture Here>
Best Practices
Best Practices – Project Planning
Involve Right People
Project Manager
• Project owner
• Decision maker
• Task & staff coordinator
Functional Owner
• Functional Impact
• New feature uptake
• Testing
• Upgrade testing
DBA
• AD utilities
• Database administration
• Technology stack updates
• Upgrade steps
IT Developer
• Functional impact
• Customization impact
• Customization development
Best Practices – Project Planning
(cont’d)
• Review appropriate documentation (Doc id: 394692.1)
to gather information on
– Upgrade process
– Tools required
– Number and types of tasks involved
– How your system and products will look in Release 12
Best Practices – Project Planning (cont’d)
• Plan to run multiple test upgrades
• Test upgrade provides
– Baseline for upgrade execution times
– Opportunity to workout any upgrade issues ahead of time
• e.g. data issues
• Plan to test key features
– e.g. Upgrade by request
• Choose hardware closely matches with that of
production during test upgrade
Best Practices – Pre-Upgrade
• Use TUMS to eliminate the tasks that are not relevant
for your system
• Use Shared file system for Multi-node
• Use Distributed AD for Multi-node
• Estimate tablespace sizes for test upgrade using Doc
id: 399362.1
Best Practices – Pre-Upgrade (cont’d)
• Modify following parameters for the duration of the
upgrade. Performance of some upgrade scripts can
be significantly improved by doing this:
– db_file_multiblock_read_count (do not set -- remove)
– _db_file_optimizer_read_count (do not set – remove)
– job_queue_processes
– parallel_max_servers
– pga_aggregate_target
– recyclebin
See Database Initialization Parameters in Chapter 1, "Planning for an Upgrade" of R12 Upgrade manual for more information on above parameters.
Best Practices – Pre-Upgrade (cont’d)
• Perform following key tasks which substantially
reduce the downtime during upgrade
– Functional tasks listed in Reducing downtime section
(Appendix E) in R12 Upgrade manual
– “Upgrade by Request” section (Appendix G) in R12 upgrade
manual
• Perform following tasks in advance to reduce
extended downtime
– Convert to Multi Org
– Convert to OATM
– Upgrade database to 10.2.0.3
Best Practices – Pre-Upgrade (cont’d)
• Gather statistics before upgrade using Gather
schema statistics concurrent program
– Use Gather Auto option if your DB is already at 10g
• Record timing for each step during test upgrade
• Make sure you have good backup before R12
upgrade and also before database upgrade
Best Practices – Pre-Upgrade (cont’d)
• Add PL/SQL no compile option in R12 upgrade driver
to save time during upgrade
– Add “extension plsql_no compile yes” line in upgrade driver
file to enable PL/SQL no compile option
extension patch_type software base
extension plsql_no compile yes
extension patchinfo maintpack 12.0.0
– Saved 3 to 4 hours of upgrade downtime during internal test
upgrades
Best Practices – Running Upgrade
• Choose proper batchsize and number of workers for AutoPatch during upgrade– For a 24 CPU database server, following parameters were used for AutoPatch on internal environment
• Batchsize=10000
• Workers=30
• To determine optimal number of workers, test with the following goals:– Between 1*CPUs and 1.5*CPUs
– Average IO response times below 10-15 milliseconds
– Average CPU usage below 100%
Best Practices – Post-Upgrade
• Make sure you reset the following init.ora parameters
after completion of R12 upgrade driver
– recyclebin
– parallel_max_servers
– job_queue_processes
• Merge all the NLS patches and apply them as single
merged patch
• Isolate post upgrade concurrent programs to a
separate manager queue as mentioned in the best
practices Doc id: 399362.1
<Insert Picture Here>
GSI Upgrade Overview
and Experiences
GSI Upgrade Overview and Experiences
• Introduction
• Overview of GSIAP Architecture
• Overview of GSIAP Statistics
• Applications Supported by AIT
• Project Planning
• Pre-Upgrade Planning
• Running the Upgrade
• Post-upgrade Planning
• Keys to Success of GSI R12 Project
Introduction
• Applications IT has a dual role – Responsible for implementing and supporting application solutions, which allow Oracle to run its business and day-to-day operations.
– Uptake new Oracle releases (such as R12) very early in the release cycle and work with Product Development to resolve issues.
• This presentation shares our experiences from the GSI R12 project.
• The experiences highlighted are appropriate for Oracle’s internal implementation and therefore may not be applicable for all upgrades.
global-DBI.
oraclecorp.com
Dell 1950
2x2.33Ghz 16G
DBI
SAN
Giga Ethernet
Giga EthernetGiga Ethernet
GSI Production Architecture - Austin Data Center
= Linux Middle Tier
= Database Server
= Storage
Intranet Load Balancing
BigIP
DMZ
Intranet Load Balancing
BigIP
iStore
iRecruitment
suppliers.com
Dell 1950
2x2.33Ghz 16G
Firewall
EMC DMX3000
Oracle RDBMS
Oracle Home Code Tree
global-forms.
oraclecorp.com
Dell 1950
2x2.33 Ghz 16G
Forms
global-
ebusiness.
oraclecorp.com
Dell 1950
2x2.33Ghz 16G
SSA
global-erp.
oraclecorp.com
Dell 1950
2x2.33Ghz 16G
ERP
global-HRMS.
oraclecorp.com
Dell 1950
2x2.33Ghz 16G
HRMS
Concurrent
Manager
Dell 1950
2x2.33Ghz 16G
Internet Load Balancing
BigIP
10G RAC (10.2.0.3)
Sun 25K
44x1.2Ghz 176G
SPIV processors
Sun 25K
44x1.2Ghz 176G
SPIV processors
Sun 25K
44x1.2Ghz 176G
SPIV processors
Sun 25K
44x1.2Ghz 176G
SPIV processors
AGSIDB01 AGSIDB02 AGSIDB03 AGSIDB04
11.5.10Application code
tree
Appl_top / Custom
NAS
Mid Tier VLAN
ssologin-ap.
oracle.com
oidap.us.oracle
.com
Dell 1950
2x2.33Ghz 16G
SSO /
OID
Ad hoc AIT
support
access
Dell 2650
2x2.8Ghz 6G
Global Single Instance Data Statistics before R12 upgrade
10 Languages20 Years of SLA Data
286,000 Projects10.5 Terabytes
382,000 Vendors
1.27 Million People
665 Sets of Books
104 Operating Units
25.8 Billion rows of data
Data Statistics for Global Single Instance – May 2007
(11.5.10.CU2)
619 Million GL lines
5.7 Million Sales Orders
15 Million AP Invoices
60 Million AR Invoice lines
9.3 Million AR Invoices
Applications Supported by AIT
GSI R12 Project Planning
Build Test Upgrades
XBUILDS 7,9,13,19
AIT/Development
UAT Test Upgrades12.0 - AIT
12.0/RUP1-AIT
12.0/RUP2-AIT/Business
12.0/RUP3-AIT/Business
Technical Timings test
Upgrades
12.0/RUP 3Nov/Dec 07
Production Upgrade
12.0/RUP 3
January 08
GSI R12 Project PlanningProject Management and Testing Strategy
• Project team consists of overall project manager to
liaise with Technical and Functional teams
• AIT POC for modules testing, POC for business testing
• Full End to End testing
GSI R12 Project PlanningProject Management and Testing Strategy (continued)
• Weekly meetings and status reports :
– review status reports with functional testers (% of testing
completed per module, critical issues highlighted, clear status for
each module, bugs open with development )
– team meetings of core project management team
– Highlighting of status/issues to Division Executive management
– Review testing status with business
– Meeting with development to discuss status/issues
– Weekly status reports are sent to wide audience to highlight
status and issues
GSI R12 Project Planning
• Environments
– Weekly patching windows to apply patches to Build Test
environment mid-week. If no issues with patch, it’s approved and
applied to UAT environment at the weekend
• Upgrade By Request
– We migrated 8 out of 20 years of SLA data and have no specific
plans to migrate remaining 12 years of data.
– Saves on downtime and first stage of planned project to archive
old transactional data
• Convert Custom Oracle Reports to XML Publisher
– Not technically required for R12
GSI R12 Pre-upgrade PlanningTasks performed prior to production outage
• Long Term pre-upgrade Tasks– Upgrade to 10.2.0.3 in advance (check Metalink for latest RDBMS patches prior to upgrade)
– Rebuild mid tiers with OEL 4 O/S
– Prepare master appl_top in test rounds (R12 Rapidinstall & NLS )
• Short Term Pre-upgrade tasks– Apply patches for Pre-upgrade steps at 11i
– Perform Pre-upgrade tasks and reducing downtime tasks
– Analyze any stale objects prior to upgrade
– Add space based on test runs using OATM model and auto-allocate for extent size (1.3 tbytes)
GSI R12 Pre-upgrade Planning
31* 13gb datafileAPPS_TS_TX_IDX
2 * 4gb datafileAPPS_TS_ARCHIVE
Tablespace Name Size
48 * 13gb datafileAPPS_TS_TX_DATA
10 * 4gb datafileAPPS_TS_SUMMARY
1 * 4gb datafileAPPS_TS_TOOLS
1 * 4gb datafileAPPS_TS_INTERFACE
1 * 4gb datafileAPPS_TS_MEDIA
1 * 4gb datafileAPPS_TS_NOLOGGING
1 * 4gb datafileAPPS_TS_SEED
1 * 4gb datafileAPPS_TS_QUEUES
GSI R12 Pre-upgrade Planning
• Transition pre-upgrade tasks– Archive log (turn off or monitor space usage)
– Disable flashback DB if enabled
– Setup space monitoring for during the upgrade
– Use “upgrade” listener rather than production listener and shutdown production listener
– Review and disable runtime production monitoring (eg EM Monitors, kill scripts for “runaway” processes, long running sessions)
– Modify init.ora parameters. In test rounds, modify init.ora parameters to reflect production
GSI R12 Pre-upgrade Planning
REMOVE_db_file_optimizer_read_count
REMOVEdb_file_multiblock_read_count
16 gbytespga_aggregate_target
8log_archive_max_processes
80parallel_max_servers
Init.ora parameter changes for R12 upgrade value
16_db_handles_cached (for ar120trxl.sql)
32010240 log_buffer
40job_queue_processes
Falsedb_block_checking
OffRecyclebin
GSI R12 Pre-upgrade Planning
• Transition pre-upgrade Tasks (continued)
– Put scheduled concurrent jobs on hold
– Run any data fixes required based on test runs
– Update u4440000.drv to include no compile option
– Create separate concurrent manager queue to process
concurrent programs submitted by upgrade
GSI R12 Running the Upgrade
• Number of workers – 60 workers across 4 mid tiers
• adpatch options used on main mid tier
options=nocompilejsp,noautoconfig,nocopyportion,
nogenerateportion
batchsize=10000, workers=60, localworkers=15
adctrl distributed=y on other 3 mid tiers
• Monitor space in new R12 tablespaces, session waits,
alert log, archive space
GSI R12 Post-upgrade Planning
• Run hrglobal before NLS (Note:145837.1 & 414434.1)
• Merge standard patches to save time
• Reset init.ora parameters to original pre-upgrade values except for
db_file_multiblock_read_count and _db_file_optimizer_read_count
and restore sga and pga to original value in test runs
• DBA steps (start up 4 nodes,FNDCPASS,enable flashback) before
testing
• Apply merged NLS D Driver in parallel with US Language Testing
• Migrate CUSTOM library (custom.pll)
• Functional post-upgrade steps/verification
• Take concurrent scheduled jobs off-hold and ensure production
monitoring is re-enabled.
Keys to Success of GSI R12 Project
• Detailed project planning and extensive
communication
• Multiple test upgrades
• Comprehensive functional testing and preparation for
impact of new features
• Completion of pre-upgrade tasks in advance
– Long Term (RDBMS, Architecture)
– Short Term
– Transition
<Insert Picture Here>
Summary
Summary
In this presentation, we have discussed following
topics:
• Overview of R12 Upgrade
• Best Practices
• GSI Upgrade Overview and Experiences
For More Information
search.oracle.com
or
http://blogs.oracle.com/schan
E-Business Suite Technology Stack Blog