Document: Installation_RAC10GR2_QFS.odt Author: Michael Ahrens, SUN Microsystems GmbH SUN Microsystems GmbH Oracle RAC on Sun Cluster and QFS Author: Michael Ahrens SUN Microsystems GmbH Created 07.06.2006 12:00:00 Last change: 09/12/2006 10:31:28 vorm. Version: 0.1 Page 1 of 90
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
SUN Microsystems GmbH Oracle RAC on Sun Cluster and QFS
Author: Michael AhrensSUN Microsystems GmbH
Created 07.06.2006 12:00:00
Last change: 09/12/2006 10:31:28 vorm.
Version: 0.1
Page 1 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Table of ContentsTable of Contents......................................................................................................................................................2Document Info...........................................................................................................................................................4
Solaris Installation.....................................................................................................................................................6Solaris 10 u1......................................................................................................................................................6Setup slvm.........................................................................................................................................................6EIS-CD..............................................................................................................................................................7QFS 4.5..............................................................................................................................................................7
Cluster Installation....................................................................................................................................................8SunCluster 3.1U4..............................................................................................................................................8Setup IPMP groups on all nodes for all public network interfaces.................................................................11Modify NVRAM parameter:...........................................................................................................................12Create Users and Groups.................................................................................................................................12Check SSH daemon.........................................................................................................................................12Setup SSH for oracle.......................................................................................................................................12Setup userenv for oracle..................................................................................................................................13Install the RAC/OPS Service from the CD.....................................................................................................14Install udlm......................................................................................................................................................14Workaround Bug.............................................................................................................................................15Set kernel parameters......................................................................................................................................17Create cluster resourcegroup for rac...............................................................................................................17Create multi-owner disk set.............................................................................................................................18Configure QFS software on all nodes.............................................................................................................19Create global mountpoint and vfstab entries on all nodes..............................................................................20Create cluster resourcegroup for qfs...............................................................................................................20Last Systemanalyse.........................................................................................................................................22
Install Clusterware 10.2.0.1.....................................................................................................................................24Run installer from ora-rac-3............................................................................................................................24Running vipca.................................................................................................................................................34Check Cluster Status.......................................................................................................................................40Install crsstat.sh...............................................................................................................................................41
Install Database Home 10.2.0.1..............................................................................................................................42Run installer from ora-rac-3............................................................................................................................42
Install Management Agent......................................................................................................................................51Run installer from ora-rac-3............................................................................................................................51
Install Patchset 10.2.0.2 on CRS Home..................................................................................................................60Stop all oracle processes.................................................................................................................................60Install binaries.................................................................................................................................................60Restart Custerware..........................................................................................................................................65
Install Patchset 10.2.0.2 on DB Home....................................................................................................................66Install binaries.................................................................................................................................................66
Install Interim Patch 5117016 on DB Home...........................................................................................................71Install with opatch...........................................................................................................................................71
Cluster Optimiziation..............................................................................................................................................78Apply Brownout Patches.................................................................................................................................78Set the CSS misscount to match the service level...........................................................................................78Run CSS in Real Time....................................................................................................................................78Set the check_interval for VIP addresses........................................................................................................78
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Document Info
Content
This is an installation summary of the installation and patch process of the RAC environment. It is based on the official procuct documentation provided by SUN and Oracle and cannot replace those doc-uments. The OS and Clusterinstallation was strongly based on EIS done.
Page 4 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Hardware Installation
Architecture
The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns from a SAN and they are mirrored with SLVM/OBAN, filesystem is QFS 4.5. The clusterinterconnect is realized with four switched gigabitethernet connections.
Drawing
Page 5 of 90
4 x Interconnect
Storage A
Oracle RAC Cluster Node: ora-rac-3
Oracle RAC ClusterNode: ora-rac-4
Storage B
IPMP Groupipmp0
IPMP Groupipmp0
MpxIO MpxIO
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Solaris Installation
Solaris 10 u1
Solaris 10 u1 SUNWCXall is installed.
• Do not forget the 512Mb (SC3.1 strikly recommended) partition for /globaldevices
Mirrored by SLVM, on every clusternode another metadevicename for this filesystem must be used.
ora-rac-3 d30
ora-rac-4 d35
• For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas.
• Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre server/storage)
change the home of root
#mkdir /root
#vi /etc/passwd
root:x:0:0:Super-User:/root:/sbin/sh
logout,login
Setup slvm
#cat /etc/lvm/md.tab
mddb1 -c 3 /dev/dsk/c3t0d0s7
d10 -m d11
d11 1 1 /dev/dsk/c3t0d0s0
d12 1 1 /dev/dsk/c3t1d0s0
d20 -m d21
d21 1 1 /dev/dsk/c3t0d0s1
d22 1 1 /dev/dsk/c3t1d0s1
d30 -m d31
d31 1 1 /dev/dsk/c3t0d0s6
d32 1 1 /dev/dsk/c3t1d0s6
Page 6 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
EIS-CD
Setup EIS environment
cd /cdrom/eis-dvd/sun/install
./setup-standard.sh
Install actual Solaris 10 patches
cd /cdrom/eis-dvd/patch/10
unpack-patches
cd /tmp/10
./install-patches
Analyse basic systemstatus
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-stall all neccessary patches.
QFS 4.5
cd /tmp/qfs_packages
pkgadd -d .
choose both qfs packages
install Patch: 122807-04
Page 7 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Additionally for Solaris10 configure /etc/inet/ipnodes
Enter all cluster nodes and all the logical Hosts.
Do not configure cluster nodes as routers!
Set up file /etc/defaultrouter scinstall will touch file /etc/notrouter per default.
defaultrouter = 192.168.10.254
When using supported network adapters which use the *ce* driver for private transport, insert into file /etc/system:
set ce:ce_taskq_disable=1
To enable MpxIO on Solaris 10 start
stmsboot -e before SunCluster will be installed !!
Do not run this command if SunCluster is already installed.
For safety reasons clean the devices with:
devfsadm -C on all nodes.
Notice: For use of VLANs for the cluster transport:
• one separated VLAN per cluster transport,
• no network traffic should be possible between the different cluster transport VLANs,
• VLANs should behave like separated physical segments.
Page 8 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Configure
#ndd -set /dev/ip ip_strict_dst_multihoming 0
You should do this in a startscript. You can use S68nettune from EIS-DVD.
See also SRDB 73036
Copy the SunCluster Software from the EIS-DVD to the disk and bunzip and untar this file. /cdrom/eisdvd/suninternal/progs/SunCluster/3.1_U4/suncluster_sc31u4_sparc_EIS.tar.bz2
Edit XML-File and remove the Explorer entries, because within the software is a very old version of the explorer.
vi ./suncluster_sc31u4_sparc/Solaris_sparc/.install/config/PPXMLS/SunCluster.xml
<description>"Explorer is required for NC to be installed"</description>
<lowest_compat_version_noupgrade>
<versionInfo versionNo="4.2"/>
</lowest_compat_version_noupgrade>
</forward_dependency>
Explorer,
Use the Java ES installer for package installation.
(Optional): To use the GUI of the installer
be sure that the DISPLAY is set correctly.
# ./installer
do NOT install JES Agents at this time !
Choose 'Configure Later'
'Configure Now' does not work this time!
Install/Establish SunCluster 3.1U4 on ora-rac-3 using scinstall.
Select menu 1)
Interactive Q+A
(Partition for /globaldevices required.)
If you not install package SUNWescom disable the scsymon service:
# svcadm -v disable /system/cluster/scsymon-srv
Page 9 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Install Sun Cluster patches: Note the structure under .../sun2/patch/SunCluster3.x/ ..3.1 (containing sub-directories, 8, 9, 10, 9_x86, 10_x86) ..agents (containing sub-directories common, 8, 9, 10, 10_x86)
the following core patch is required for SC3.1U4:
• 120500-04 Sun Cluster 3.1 Core Patch for Solaris 10
Page 10 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Setup IPMP groups on all nodes for all public network interfaces.
At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g: IPMP group active-active with interfaces ce0 and ce4 with one physical IP.
ora-rac-3
Entry of /etc/hostname.ce0 :
ora-rac-3-1 netmask + broadcast + group ipmp0 up \
addif ora-rac-3 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-3-2 netmask + broadcast + group ipmp0 deprecated -failover up
ora-rac-4
Entry of /etc/hostname.ce0 :
ora-rac-4-1 netmask + broadcast + group ipmp0 up \addif ora-rac-3 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-4-2 netmask + broadcast + group ipmp0 deprecated -failover up
If the defaultrouter is NOT 100% available please read infodocs
70060 and 70062.
Notes:
• Do not use test IP for normal applications
• Test IP for all adapters in the same IPMP group must belong to a single IP subnet
• You need an addtional IP for each logical host.
• In the /etc/default/mpathd file, the value of
TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)
Common actions for filesystems and boot devices:
Set localonly flag for the did device of the root mirror disk:
• make sure only local node in node list:
# scconf -pvv | grep dsk/d<N>• if other nodes in node list, remove them:# scconf -r -D name=dsk/d<N>,nodelist=<other_phy_host>• set localonly on root mirror disk:# scconf -c -D name=dsk/d<N>,localonly=true
NOTE: To find out the did device use scdidadm -l command!
Enter global filesystems into /etc/vfstab on all nodes. Recommendation:
mount archivelog destinaton FS as an QFS Filesystem for each instance not as an global filesystem.
Page 11 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
See later in this document for qfs. Create all mountpoints on all nodes.
Add global and logging option in /etc/vfstab if you use UFS.
Modify NVRAM parameter:
Set boot-device & diag-device to both sides of the mirror.
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
export ORACLE_SID
export ORACLE_HOME
EDITOR=vi
export EDITOR
if [ -t 0 ]; then
stty intr ^C
fi
# stty erase ^?
alias sql='sqlplus / as sysdba'
alias crsi='/oracle/product/crs/bin/crs_stat -t'
alias xxx=". ~/set_oracle_env.sh"
alias oh='cd $ORACLE_HOME'
alias al='vi /oracle/admin/ZDA/bdump/alert_${ORACLE_SID}.log'
echo "Tip: Enter xxx to change your current setting"
Install the RAC/OPS Service from the CD
Via pkgadd on all nodes. You always need SUNWscucm SUNWudlm SUNWudlmr.
• For SVM you require SUNWscmd.
• SUNWschwr is needed for Shared QFS for Hardware Raid Support.
Install udlm
Remark: Don't use the version from the EIS DVD, for Oracle 10G the asynchronous version is re-quired, on the EIS DVD is only the synchronous version available. See chapter „Prepair Oracle Installa-tion“ for the Installationmedias.
#cd /install/oracle/stage/cl/racpatch
#uncompress ORCLudlm.tar.Z;tar xf ORCLudlm.tar
#pkgadd -d .
#The following packages are available:
1 ORCLudlm Oracle UNIX Distributed Lock Manager
(sparc) Dev Release 06/11/04, 64bit 3.3.4.8 reentrant, async libskgxn2.so
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
Page 14 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Workaround Bug
The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.
mkdir -p /usr/local/bin
ln -s /usr/bin/ssh /usr/local/bin/ssh
ln -s /usr/bin/scp /usr/local/bin/scp
If you use Solaris Secure Shell (sshd) with enabled X11 forwarding and udlm you should read Sun Al-ert 57428.
Configure 'udlm.port' and 'udlm.num_ports' in /opt/SUNWudlm/etc/udlm.conf
# Modified by rac_udlm_init on Tue May 9 09:14:16 CEST 2006
#
# ident "@(#)udlm.conf.template 1.5 03/04/15 SMI"
#
# Copyright 1999-2003 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# This file has been created by SUNW.rac_udlm resource type
If you are not able to register SUNW.qfs then create the link:
# cd /usr/cluster/lib/rgm/rtreg
# ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs
Set ownership and permissions:
chown oracle:dba /oradata /oraarch
chmod 755 /oradata /oraarch
Page 21 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Last Systemanalyse
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and in-stall all neccessary patches.
Page 22 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Prepare Oracle Installation
Share staging area
All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle and can be mounted on every node if needed.
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Create Database ZDA
Run DBCA
>> ora-rac-3
/oracle/product/db/bin/dbca
Page 74 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 75 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Page 76 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Password: oracle
Page 77 of 90
Document: Installation_RAC10GR2_QFS.odtAuthor: Michael Ahrens, SUN Microsystems GmbH
Cluster OptimiziationOverview and best practices for setting up a "zero brownout" configuration to achieve maximum high availability. More info in note 369101.1.
Note: Patch 5024639 is not yet available and skipped therefore.
Apply Brownout Patches
This patches are installed by using OPatch on DB and ASM Home:
5186679 (This became patch 4960705)
5186674 (This became patch 5016142)
5127434
5127472
5127482
5176453
Set the CSS misscount to match the service level
1. Shut down the clusterware stack on all nodes but one
2. On the remaining node, run as root:
$ORA_CRS_HOME/bin/crsctl set css misscount 5
$ORA_CRS_HOME/bin/crsctl set css disktimeout 200
3. Reboot the nodes for this to take affect.
4. Confirm the new values for the CSS parameters
$ORA_CRS_HOME/bin/crsctl get css misscount
$ORA_CRS_HOME/bin/crsctl get css disktimeout
Run CSS in Real Time
1. On each node as root use, crsctl set css priority 4
2. Restart each node in series, using crsctl stop crs followed by crsctl start crs
Set the check_interval for VIP addresses
>> ora-ra-3 as root
1. Generate the resource profile for the VIP resource