Upgrade 11gR2 to 12cR1 Clusterware

Upgrade11gR2 to12cR1 Clusterware

Presenter : Nikhil Kumar

2

WHO AM I?Nikhil Kumar (DBA Manager)

6 Years of Experience in Oracle Databases and Apps.

Oracle Certified Professional Oracle 9i and 11g.

Worked on Mission critical Databases (Telecom, Financial ERP, Manufacturing and Government Domain).

Member and Speaker of AIOUG-North India Chapter (http://www.aioug.org/aiougnichapter.php)

Contact me at [email protected]

http://nikhildbasavvy.wordpress.com/

https://www.facebook.com/groups/OracleTechSavvy/

Twitter: @nikhil0028us

http://www.slideshare.net/nikhil0028us

in.linkedin.com/pub/nikhil-kumar/1b/a74/350/

3

AGENDA

Introduction of Clusterware

Pros and Cons of Upgrade

Prerequisite

Traditional cluster Vs Flex Cluster

Clusterware Upgrade

Recovering from rootupgrade.sh failure

Clusterware Downgrade

Tips to monitor and improve the RAC environment

Oracle Clusterware (Platform on Platform)

Oracle clusterware is infrastructure which provides platform to Oracle database run on shared mode (Active- Active).

Oracle Clusterware is acting as platform on OS platform to provide database availability in shared mode.

5

Traditional Cluster Vs Flex Cluster

6

Pros and Cons of Upgrade

Pros(Requirement)

New Features and components

Support from Oracle

Bug fixes

Sometimes requirement of advance application

Database upgrade requirement (Database version cannot be greater than Clusterware version).

Cons

Risk of failure during upgrade

Risk of long outage in 24/7 applications

Lack of skills to newer version

Legacy Application

7

Prerequisite

Backup of existing Clusterware and OCR

Sufficient Space in installation mountpoint

GRID Home directory (Out of place Upgrade)

Unset Oracle Variable(ORACLE_HOME,ORACLE_BASE and TNS_ADMIN etc)

OS Groups

Kernel Parameters and Packages

Root or sudo user access

11.2.0.2.3 minimum clusterware version required to upgrade on 12.1.0.1

8

Cluster Overview

Two Node cluster

Operating System version RHEL 6.4

Cluster and database software version 11.2.0.4.0

Cluster Name: NIOUG

SCAN: racnode.linuxdc.com

Raw Disk size -- 10 Luns

Diskgroups (OCR)

Cluster Overview

10

Upgradion of clusterware from 11.2.0.4 to 12.1.0.1

11

Select Installation Option

12

Cluster Overview

13

Grid Infrastructure Node Selection

14

Grid Infrastructure Management Repository Option

15

Grid Management database MGMTDB is new database instance which is used for storing Cluster Health Monitor (CHM) data.

Instance is created on first node only.

Get failover to surviving node if existing node crash.

Databases files for repository database are stored in same location as OCR/Voting disk

[oracle@racnode1 ~]$ srvctl config mgmtdb

$srvctl status mgmtdb

Database is enabled Instance -MGMTDB is running on node racnode1

[oracle@racnode1 ~]$ oclumon manage -get MASTER

Master = racnode1

[oracle@racnode1 ~]$ oclumon manage -get reppath

CHM Repository Path = +OCR/_MGMTDB/DATAFILE/sysmgmtdata.259.85381046

[oracle@racnode1 ~]$ oclumon version

Cluster Health Monitor (OS), Version 12.1.0.1.0 - Production Copyright 2007, 2013 Oracle. All rights reserved.

For command reference

http://docs.oracle.com/database/121/CWADD/troubleshoot.htm#CWADD92340

16

Group Information

17

Installation Location

18

Root Script Executionecution

19

Root Script Execution

20

Select batches for nodes

21

Summary of configuration

22

Upgradation in process

23

Upgradation in process

24

Running rootupgrade.sh in background

25

Click “Continue” to run rootupgrade.sh on first node

26

rootupgrade.sh on first node

27

Press “Ok” to run rootupgrade.sh on second node

28

Recovering from rootupgrade.sh failure

Changing the First Node for Upgrade

If the first node becomes inaccessible, you can force another node to be the first node for installation or upgrade. run the following command on another node using the -force option:

rootupgrade.sh -force -first

Problems before rootupgrade.sh failure : MOS ID1056322.1

Problem running rootupgrade.sh : MOS ID 1364947.1

29

Upgradation in-process

30

Management database creation issue

31

Upgradation Complete

32

Checking Clusterware Services

ps –ef| grep d.bin

crsctl stat res –t

33

Downgrade Clusterware from 12.1.0.1 to 11.2.0.4

34

Downgrade Clusterware from 12.1.0.1 to 11.2.0.4

1.On all remote nodes, use the command

# /u01/app/12.1.0/grid/crs/install/rootcrs.sh –downgrade (This command will shutdown the Grid clusterware stack and restore the old Clusterware home files)

2. After the rootcrs.sh -downgrade script has completed on all remote nodes, on the local node use the command

# /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade –lastnode ( This command will restore the old OCR file)

This script downgrades the OCR. If you want to stop a partial or failed Oracle Grid Infrastructure 12c Release 1 (12.1) installation and restore the previous release Oracle Clusterware, then use the -force flag with this command.

Update the registry:-

$ cd /u01/app/12.1.0/grid/oui/bin ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/12.1.0/grid

$ cd /u01/app/12.1.0/grid/oui/bin $ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/crs

Start the clusterware from old clusterware home on all nodes:-

#/u01/app/11.2.0/grid/bin/crsctl start crs

35

Downgrade Clusterware from 12.1.0.1 to 11.2.0.4

36

Downgrade…Behind the scene

014-07-24 18:25:02: Successfully execute 'acfsroot install' from older home

2014-07-24 18:25:02: Restore init files

2014-07-24 18:25:02: restore init scripts

2014-07-24 18:25:02: copy "/u01/app/11.2.0/grid/crs/init/init.ohasd" => "/etc/init.d/init.ohasd"

2014-07-24 18:25:02: copy "/u01/app/11.2.0/grid/crs/init/ohasd" => "/etc/init.d/ohasd"

2014-07-24 18:25:02: leftVersion=11.2.0.4.0; rightVersion=11.2.0.3.0

2014-07-24 18:25:02: [11.2.0.4.0] is higher than [11.2.0.3.0]

2014-07-24 18:25:02: Remove all new version related stuff from /etc/oratab

2014-07-24 18:25:02: Copying file /etc/oratab.new.racnode1 to /etc/oratab

2014-07-24 18:25:02: copy "/etc/oratab.new.racnode1" => "/etc/oratab"

2014-07-24 18:25:02: Removing file /etc/oratab.new.racnode1

2014-07-24 18:25:02: Removing file /etc/oratab.new.racnode1

2014-07-24 18:25:02: Successfully removed file: /etc/oratab.new.racnode1

2014-07-24 18:25:02: Removing the checkpoint file /u01/app/grid/crsdata/racnode1/crsconfig/ckptGridHA_racnode1.xml

2014-07-24 18:25:02: Removing file /u01/app/grid/crsdata/racnode1/crsconfig/ckptGridHA_racnode1.xml

2014-07-24 18:25:02: Successfully removed file: /u01/app/grid/crsdata/racnode1/crsconfig/ckptGridHA_racnode1.xml

2014-07-24 18:25:02: Successfully downgraded Oracle Clusterware stack on this node

37

Be Careful.. While clusterware downgrade

Running rootupgrade.sh in wrong sequence during downgrade of cluster could lead to below error.

38

Tips to monitor and improve the Cluster environment.There are so many tool and which can improve your configuration of RAC.

1.ORAchk Tool: Proactively scans for the most impactful problems across the various layers of your stack. It perform check on OS Level, Cluster level, database level and Network Level and provide solution according to that.

Also refer ORAchk - Oracle Configuration Audit Tool (Doc ID 1268927.2)

2.OSWatcher and/or CHM Cluster health monitor :- These tools are used to monitor the operating and clusterware level process and record them in log according to your retention policy.

Also refer OSWatcher (Includes: [Video]) (Doc ID 301137.1)

3. RAC :- RAC comes with few more AWR report functionality.

awrgrpt:- It gives you the detail information of all nodes of the cluster in single report.

awrgdrpt:- Its gives you the difference report b/w 2 awr reports to deal with performance issue.

.

39

40