10.2.0.5 Patch Set Apply on Oracle Clusterware, ASM and RDBMS on 2-Node RAC Linux x86_64 Version: V0.1 By Raheel H Syed 10g OCP Certified Blog: http://raheeldba.wordpress.com Twitter: https://twitter.com/raheelhsyed LinkedIn: http://in.linkedin.com/pub/raheel-hashmi-
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
10.2.0.5 Patch Set Apply on
Oracle Clusterware,
ASM and RDBMS on
2-Node RAC Linux x86_64
Version: V0.1
By
Raheel H Syed10g OCP CertifiedBlog: http://raheeldba.wordpress.comTwitter: https://twitter.com/raheelhsyed LinkedIn: http://in.linkedin.com/pub/raheel-hashmi-syed/33/34a/226
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Pre-installation Tasks
RACNODE1 and RACNODE2 have separate homes for ASM and Oracle Home and CRS Home. All homes will be patched on both the nodes
Existing CRS_HOME, ASM_HOME & ORACLE_HOME version is 10.2.0.4
+ASM1:/app/asm
PROD1:/app/oracle/10.2.0/db_1
CRS home /app/crs/home_1
1. Backup CRS, ASM, DB Home and inventory for both the nodes
- Use tar to backup the homes as ‘ tar –cvf ’
2. Check opatch lsinventory on CRS, ASM and Oracle Home. Save the output.
3. Check cluster health, crs activeversion and software version on both the nodes and save the output.
$ crsctl check crs
$ crsctl query css activeversionCRS active version on the cluster is [10.2.0.4.0]
$ crsctl query css software versionCRS software version on node [racnode1] is [10.2.0.4.0]
4. Upgrading Installations that use Java Virtual Machine or Oracle interMedia
- If the database uses JVM or Oracle interMedia, then install the oracle database 10g Products installation type from 10g Companion CD before installing 10.2.0.5 Patchset. MOS ID: 293658.1
5. Update Oracle Time Zone Definitions
(A) When patching from a lower 10.2 release or upgrading from 10.1.0.x directly to 10.2.0.5, check the current version of the Oracle time zone definitions with the following command:
SELECT version FROM v$timezone_file;
>>> If this query reports version 4, no action is required.
Raheel H Syed 2
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
>>> If this reports a version lower or higher than 4, see My Oracle Support document 1086400.1 Actions for the DSTv4 update in the Release 10.2.0.5 patchset.
(No Action required in our case as version is 4)
6. chmod -R +w CRS_HOME/inventory/Templates/*
7. Save all db links - Incase downgrade is required you can drop the db links and recreate after if downgrade is required
spool dblinks_pre_upgrade.lstSELECT'create '||DECODE(U.NAME,'PUBLIC','public ')||'database link '||CHR(10)||DECODE(U.NAME,'PUBLIC',Null, U.NAME||'.')|| L.NAME||chr(10)||'connect to ' || L.USERID || ' identified by '''||L.PASSWORD||''' using ''' || L.host || ''''||chr(10)||';' TEXTFROM sys.link$ L,sys.user$ UWHERE L.OWNER# = U.USER#;spool off
8. Check invalid objects
spool invalid_pre_upgrade.lstselect substr(owner,1,12) owner,substr(object_name,1,30) object,substr(object_type,1,30) type, status fromdba_objects where status <> 'VALID';spool off
Run utlrp.sql and re-query invalid objects
The last queries (invalid objects check) will lists all objects that cannot be compiled before upgrade in file 'invalid_pre.lst'
9. Create pfile from spfile AND backup tnsnames.ora and listener.ora
10. Check AUD$ table, if auditing is enabled then upgrade performance will be affected
Raheel H Syed 3
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
SELECT count (*) FROM SYS.AUD$;
count (*)--------- 0
MOS Id: 979942.1
(No action required in our case as NO auditing is enabled)
11. Verify database components, status and version from dba_registry. Save the output.
set pages 500set lines 120col COMP_NAME for a40select comp_name, status, version from dba_registry;
12. Backup the database. Using RMAN, backup the full database and archive log in compressed format to disk. Though we had tape backups configured, we opted for disk backups as backup completion time to tapes varied to 4-6 hours. Disk backup completed in 2 hours.
#!/usr/bashexport NLS_DATE_FORMAT="DD-MON-YY HH24:MI:SS"rman target / log /tmp/before_upgrade_full_db.log<<EOFrun{allocate channel ch001 device type disk;allocate channel ch002 device type disk;allocate channel ch003 device type disk;allocate channel ch004 device type disk;BACKUP AS COMPRESSED BACKUPSET DATAFILE 1,2,3,4,5,22,23,24 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 25,26,28,29,31,32,33,34 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 35,36,37,39,40,42,43,44 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 45,46,47,48,49,50,51,52 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 53,54,55,56,58,59,60,61 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 62,63,64,65,66,67,68,69 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';
Raheel H Syed 4
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
BACKUP AS COMPRESSED BACKUPSET DATAFILE 70,71,72,73,74,75,76,77 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 78,79,80,81,82,83,84,85 format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 6,7,8,9,10,11,12,13 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 14,15,16,17,18,19,20,21 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';BACKUP AS COMPRESSED BACKUPSET DATAFILE 27,30,38 format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';backup as backupset format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.ctl' current controlfile tag='Before_upgrade_CTL';sql 'alter system archive log current';sql 'alter system archive log current';sql 'alter system archive log current';BACKUP AS COMPRESSED BACKUPSET format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.arc' archivelog all tag='Before_upgrade_ARCH';backup as backupset format '/app1/prod1/before_upgrade/%d_before_upgrade_AFTER_ARCH_%s_%p_%c_%t.ctl' current controlfile tag='Before_upgrade_CTL';release channel ch001;release channel ch002;release channel ch003;release channel ch004;}quitEOF
Note: I have explicitly specified datafiles for each compressed backupset, this is because there was not enough space at /app mount point.
Database size 1230 GB and compressed backup size on disk 139 GB.
Upgrade Clusterware1. Resolve any outstanding unresolved distributed transactions:
SQL> select * from dba_2pc_pending;
If this returns rows then run the following
SQL> select local_tran_id from dba_2pc_pending;SQL> execute dbms_transaction.purge_lost_db_entry ('<input_id>'); SQL> commit;
Raheel H Syed 5
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
2. Disable the jobs that run from scheduler before upgrading the database.
Downtime starts here.
Note: In our case all the production jobs were scheduled from DBMS_SCHEDULER hence we have disabled the jobs.
3. Stop database, ASM and nodeapps on both the nodes.
4. Download and unzip patch p8202632_10205_Linux-x86-64.zip
5. We have used VNC viewer to install this patch set, most of the Linux system comes with vnc service running on the system. Please check with your SA if it’s not running. To check if vnc service is running on the system or not, issue the following command
$ ps –ef | grep vnc
If the output shows …… oracle ….. <your hostname>:<number> then it means you have vnc session running under oracle user, eg: racnode1:1 or racnode1:2
If VNC service is not running then you can do this, it worked for me
As ‘Oracle’ user:
$ cd /usr/bin
$ ls –lrt vnc*
$ ./vncserver
It will ask for password, provide the password and verify.
6. CD to Disk1 and Invoke runInstaller and select CRS HOME from the path. Follow the instructions as displayed on the screen.
OR
Execute the following to ignore pre-requisite checks of parameters (If required)
$ ./runinstaller –ignoreSysPrereqs
Raheel H Syed 6
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(A)
(B) Select CRS_HOME
Raheel H Syed 7
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(C) Click next by default all the nodes in cluster are selected as below
(D) Warning: Backup /etc/sysctl.conf and modify fs.file_max=101365 (see below)
Raheel H Syed 8
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(E)
(F) stop crs as ROOT user; As ROOT user run CRS_HOME/install/root102.sh
Raheel H Syed 9
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
7. Issues faced during upgrade:-
- Prerequisite check failed as fs.file-max = 655367, required value 101365.
Asked SA Admin to backup the /etc/sysctl.conf and modify the parameter.
- Error in writing to CRS_Home:/app/oracle/product/10.2.0/crs/OPatch
Change permission as $ chmod +w opatch
8. The script will automatically start crs daemon on the patched node upon completion
9. crs_stat -t (or) crs_status -t on completed node
10. Proceed to node2 and repeat.
This completes CRS upgrade to 10.2.0.5
11. Check crs softwareversion and activeversion on both the nodes. On successful upgrade crs softwareversion and activeversion should be at 10.2.0.5
$ crsctl check crs softwareversion
Raheel H Syed 10
10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
$ crsctl check crs activeversion
Upgrade ASM and Oracle Home
1. Stop database, ASM and nodeapps on both the nodes.