Oracle 10g RAC On Linux Using VMware Server
Oracle 10g RAC On Linux Using VMware Server
This article describes the installation of Oracle 10g release 2
(10.2.0.1) RAC on Linux (CentOS 4) using VMware Server with no
additional shared disk devices.
Introduction
Download Software
VMware Server Installation
Virtual Machine Setup
Guest Operating System Installation
Oracle Installation Prerequisites
Install VMware Client Tools
Create Shared Disks
Clone the Virtual Machine
Install the Clusterware Software
Install the Database Software and Create an ASM Instance
Create a Database using the DBCA
TNS Configuration
Check the Status of the RAC
Introduction
One of the biggest obstacles preventing people from setting up
test RAC environments is the requirement for shared storage. In a
production environment, shared storage is often provided by a SAN
or high-end NAS device, but both of these options are very
expensive when all you want to do is get some experience installing
and using RAC. A cheaper alternative is to use a FireWire disk
enclosure to allow two machines to access the same disk(s), but
that still costs money and requires two servers. A third option is
to use VMware Server to fake the shared storage.
Using VMware Server you can run multiple Virtual Machines (VMs)
on a single server, allowing you to run both RAC nodes on a single
machine. In additon, it allows you to set up shared virtual disks,
overcoming the obstacle of expensive shared storage.
Before you launch into this installation, here are a few things
to consider.
The finished system includes the host operating system, two
guest operating systems, two sets of Oracle Clusterware, two ASM
instances and two Database instances all on a single server. As you
can imagine, this requires a significant amount of disk space, CPU
and memory. I tried this installation on a 3.4G Pentium 4 with 2G
of memory and it failed abysmally. When I used a dual 3.0G Xeon
server with 4G of memory it worked fine, but it wasn't exactly
fast.
This procedure provides a bare bones installation to get the RAC
working. There is no redundancy in the Clusterware installation or
the ASM installation. To add this, simply create double the amount
of shared disks and select the "Normal" redundancy option when it
is offered. Of course, this will take more disk space.
During the virtual disk creation, I always choose not to
preallocate the disk space. This makes virtual disk access slower
during the installation, but saves on wasted disk space.
This is not, and should not be considered, a production-ready
system. It's simply to allow you to get used to installing and
using RAC.
Download Software
Download the following software.
CentOS 4.x
VMware Server
Oracle 10g (10.2.0.1) CRS and DB software
VMware Server Installation
For this article, I used CentOS 4.3 as both the host and guest
operating systems. Regardless of the host OS, the setup of the
virtual machines should be similar.
First, install the VMware Server software. On Linux you do this
with the following command as the root user.
# rpm -Uvh VMware-server-*.rpm
Preparing... ###########################################
[100%]
1:VMware-server ###########################################
[100%]
#
Then finish the configuration by running the vmware-config.pl
script as the root user. Most of the questions can be answered with
the default response by pressing the return key. The output below
shows my responses to the questions.
# vmware-config.pl
Making sure services for VMware Server are stopped.
Stopping VMware services:
Virtual machine monitor [ OK ]
You must read and accept the End User License Agreement to
continue.
Press enter to display it.
VMWARE, INC.
SOFTWARE BETA TEST AGREEMENT
*** Editied out license agreement ***
Do you accept? (yes/no) yes
Thank you.
Configuring fallback GTK+ 2.4 libraries.
In which directory do you want to install the mime type
icons?
[/usr/share/icons]
What directory contains your desktop menu entry files? These
files have a
.desktop file extension. [/usr/share/applications]
In which directory do you want to install the application's
icon?
[/usr/share/pixmaps]
Trying to find a suitable vmmon module for your running
kernel.
The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the
running kernel.
Do you want networking for your virtual machines? (yes/no/help)
[yes]
Configuring a bridged network for vmnet0.
The following bridged networks have been defined:
. vmnet0 is bridged to eth0
All your ethernet interfaces are already bridged.
Do you want to be able to use NAT networking in your virtual
machines? (yes/no)
[yes]
Configuring a NAT network for vmnet8.
Do you want this program to probe for an unused private subnet?
(yes/no/help)
[yes]
Probing for an unused private subnet (this can take some
time)...
The subnet 172.16.210.0/255.255.255.0 appears to be unused.
The following NAT networks have been defined:
. vmnet8 is a NAT network on private subnet 172.16.210.0.
Do you wish to configure another NAT network? (yes/no) [no]
Do you want to be able to use host-only networking in your
virtual machines?
[yes] no
Trying to find a suitable vmnet module for your running
kernel.
The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the
running kernel.
Please specify a port for remote console connections to use
[902]
Stopping xinetd: [ OK ]
Starting xinetd: [ OK ]
Configuring the VMware VmPerl Scripting API.
Building the VMware VmPerl Scripting API.
Using compiler "/usr/bin/gcc". Use environment variable CC to
override.
The installation of the VMware VmPerl Scripting API
succeeded.
Do you want this program to set up permissions for your
registered virtual
machines? This will be done by setting new permissions on all
files found in
the "/etc/vmware/vm-list" file. [no] yes
Generating SSL Server Certificate
In which directory do you want to keep your virtual machine
files?
[/var/lib/vmware/Virtual Machines] /u01/VM
Do you want to enter a serial number now? (yes/no/help) [no]
yes
Please enter your 20-character serial number.
Type XXXXX-XXXXX-XXXXX-XXXXX or 'Enter' to cancel:
ENTER-YOUR-SERIAL-NUMBER
Starting VMware services:
Virtual machine monitor [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host-only networking on /dev/vmnet8 (background) [ OK ]
NAT service on /dev/vmnet8 [ OK ]
Starting VMware virtual machines... [ OK ]
The configuration of VMware Server e.x.p build-22874 for Linux
for this running
kernel completed successfully.
#
The VMware Server Console is started by issuing the command
"vmware" at the command prompt, or by selecting it from the "System
Tools" menu.
INCLUDEPICTURE
"http://www.oracle-base.com/articles/10g/images/10gRACLinux/01-VMwareServerMenu.jpg"
\* MERGEFORMATINET
On the "Connect to Host" dialog, accept the "Local host" option
by clicking the "Connect" button.
You are then presented with the main VMware Server Console
screen.
The VMware Server is now installed and ready to use.
Virtual Machine Setup
Now we must define the two virtual RAC nodes. We can save time
by defining one VM, then cloning it when it is installed.
Click the "Create a new virtual machine" button to start the
"New Virtual Machine Wizard". Click the "Next" button onthe welcome
page.
Select the "Custom" virtual machine configuration and click the
"Next" button.
Select the "Linux" guest operating system option, and set the
version to "Red Hat Enterprise Linux 4", then click the "Next"
button.
Enter the name "RAC1" and the location should default to
"/u01/VM/RAC1", then click the "Next" button.
Select the required number of processors and click the "Next"
button.
Uncheck the "Make this virtual machine private" checkbox and
click the "Next" button.
Select the amount of memory to associate with the virtual
machine. Remember, you are going to need two instances, so don't
associate too much, but you are going to need approximately 1 Gig
(1024 Meg) to compete the installation successfully.
Accept the "Use bridged networking" option by clicking the
"Next" button.
Accept the "LSI Logic" option by clicking the "Next" button.
Select the "Create a new virtual disk" option and click the
"Next" button.
Accept the "SCSI" option by clicking the "Next" button. It's a
virtual disk, so you can still use this option even if your
physical disk is IDE or SATA.
Set the disk size to "10.0" GB and uncheck the "Allocate all
disk space now" option. The latter will make disk access slower,
but will save you wasting disk space.
Accept "RAC1.vmdk" as the disk file name and complete the VM
creation by clicking the "Finish" button.
On the "VMware Server Console" screen, click the "Edit virtual
machine settings" button.
On the "Virtual Machine Settings" screen, highlight the "Floppy
1" drive and click the "- Remove" button.
Click the "+ Add" button and select a hardware type of "Ethernet
Adapter", then click the "Next" button.
Accept the "Bridged" option by clicking the "Finish" button.
Click on the "Options" tab, highlight the "Startup/Shutdown"
setting and select the "Don't power on virtual machine" in the "On
host startup" option. Finish by clicking the "OK" button.
The virtual machine is now configured so we can start the guest
operating system installation.
Guest Operating System Installation
Place the first CentOS 4 disk in the CD drive and start the
virtual machine by clicking the "Power on this virtual machine"
button. The right pane of the VMware Server Console should display
a boot loader, then the CentOS installation screen.
Continue through the CentOS 4 installation as you would for a
normal server. A general pictorial guide to the installation can be
found here. More specifically, it should be a server installation
with a minimum of 2G swap, secure Linux disabled and the following
package groups installed:
X Window System
GNOME Desktop Environment
Editors
Graphical Internet
Server Configuration Tools
FTP Server
Development Tools
Legacy Software Development
Administration Tools
System Tools
To be consistent with the rest of the article, the following
information should be set during the installation:
hostname: rac1.localdomain
IP Address eth0: 192.168.2.101 (public address)
Default Gateway eth0: 192.168.2.1 (public address)
IP Address eth1: 192.168.0.101 (private address)
Default Gateway eth1: none
You are free to change the IP addresses to suit your network,
but remember to stay consistent with those adjustments throughout
the rest of the article.
Once the basic installation is complete, install the following
packages whilst logged in as the root user.
# From CentOS4 Disk 1
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh setarch-1*
rpm -Uvh compat-libstdc++-33-3*
rpm -Uvh make-3*
rpm -Uvh glibc-2*
cd /
eject
# From CentOS4 Disk 2
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh openmotif-2*
rpm -Uvh compat-db-4*
rpm -Uvh gcc-3*
cd /
eject
# From CentOS4 Disk 3
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh libaio-0*
rpm -Uvh rsh-*
rpm -Uvh compat-gcc-32-3*
rpm -Uvh compat-gcc-32-c++-3*
rpm -Uvh openmotif21*
cd /
eject
Oracle Installation Prerequisites
Perform the following steps whilst logged into the RAC1 virtual
machine as the root user.
The /etc/hosts file must contain the following information.
127.0.0.1 localhost.localdomain localhost
# Public
192.168.2.101 rac1.localdomain rac1
192.168.2.102 rac2.localdomain rac2
#Private
192.168.0.101 rac1-priv.localdomain rac1-priv
192.168.0.102 rac2-priv.localdomain rac2-priv
#Virtual
192.168.2.111 rac1-vip.localdomain rac1-vip
192.168.2.112 rac2-vip.localdomain rac2-vip
Add the following lines to the /etc/sysctl.conf file.
kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default=262144
net.core.rmem_max=262144
net.core.wmem_default=262144
net.core.wmem_max=262144
Run the following command to change the current kernel
parameters.
/sbin/sysctl -p
Add the following lines to the /etc/security/limits.conf
file.
* soft nproc 2047
* hard nproc 16384
* soft nofile 1024
* hard nofile 65536
Add the following line to the /etc/pam.d/login file, if it does
not already exist.
session required /lib/security/pam_limits.so
Disable secure linux by editing the /etc/selinux/config file,
making sure the SELINUX flag is set as follows.
SELINUX=disabled
Alternatively, this alteration can be done using the GUI tool
(Applications > System Settings > Security Level). Click on
the SELinux tab and disable the feature.
Set the hangcheck kernel module parameters by adding the
following line to the /etc/modprobe.conf file.
options hangcheck-timer hangcheck_tick=30
hangcheck_margin=180
To load the module immediately, execute "modprobe -v
hangcheck-timer".
Create the new groups and users.
groupadd oinstall
groupadd dba
groupadd oper
useradd -g oinstall -G dba oracle
passwd oracle
Create the directories in which the Oracle software will be
installed.
mkdir -p /u01/crs/oracle/product/10.2.0/crs
mkdir -p /u01/app/oracle/product/10.2.0/db_1
mkdir -p /u01/oradata
chown -R oracle.oinstall /u01
During the installation, both RSH and RSH-Server were installed.
Enable remote shell and rlogin by doing the following.
chkconfig rsh on
chkconfig rlogin on
service xinetd reload
Create the /etc/hosts.equiv file as the root user.
touch /etc/hosts.equiv
chmod 600 /etc/hosts.equiv
chown root:root /etc/hosts.equiv
Edit the /etc/hosts.equiv file to include all the RAC nodes:
+rac1 oracle
+rac2 oracle
+rac1-priv oracle
+rac2-priv oracle
Login as the oracle user and add the following lines at the end
of the .bash_profile file.
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1; export
ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export
LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib;
export CLASSPATH
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
Install VMware Client Tools
Login as the root user on the RAC1 virtual machine, then select
the "VM > Install VMware Tools..." option from the main VMware
Server Console menu.
This should mount a virtual CD containing the VMware Tools
software. Double-click on the CD icon labelled "VMware Tools" to
open the CD. Right-click on the ".rpm" package and select the "Open
with 'Install Packages'" menu option.
Click the "Continue" button on the "Completed System
Preparation" screen and wait for the installation to complete.
Once the package is loaded, the CD should unmount automatically.
You must then run the "vmware-config-tools.pl" script as the root
user. The following listing is an example of the output you should
expect.
# vmware-config-tools.pl
Stopping VMware Tools services in the virtual machine:
Guest operating system daemon: [ OK ]
Trying to find a suitable vmhgfs module for your running
kernel.
The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the
running kernel.
pcnet32 30409 0
Unloading pcnet32 module
Trying to find a suitable vmxnet module for your running
kernel.
The module bld-2.6.9-11.EL-i686up-RHEL4 loads perfectly in the
running kernel.
Detected X.org version 6.8.
Please choose one of the following display sizes (1 - 13):
[1] "640x480"
[2] "800x600"
[3] "1024x768"
[4] "1152x864"
[5] "1280x800"
[6] "1152x900"
[7] "1280x1024"
[8] "1376x1032"
[9] "1400x1050"
[10] "1680x1050"
[11] "1600x1200"
[12]< "1920x1200"
[13] "2364x1773"
Please enter a number between 1 and 13:
[12] 3
X Window System Version 6.8.2
Release Date: 9 February 2005
X Protocol Version 11, Revision 0, Release 6.8.2
Build Operating System: Linux 2.6.9-11.EL i686 [ELF]
Current Operating System: Linux rac1.localdomain 2.6.9-22.EL #1
Sat Oct 8 17:48:27 CDT 2005 i686
Build Date: 07 October 2005
Build Host: x8664-build.home.local
Before reporting problems, check http://wiki.X.Org
to make sure that you have the latest version.
Module Loader present
OS Kernel: Linux version 2.6.9-22.EL
([email protected])
(gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #1 Sat Oct 8
17:48:27 CDT 2005 P
Markers: (--) probed, (**) from config file, (==) default
setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??)
unknown.
(++) Log file: "/tmp/vmware-config0/XF86ConfigLog.3674", Time:
Thu Apr 13 21:17:37 2006
(++) Using config file:
"/tmp/vmware-config0/XF86Config.3674"
X is running fine with the new config file.
(WW) VMWARE(0): Failed to set up write-combining range
(0xf0000000,0x1000000)
Starting VMware Tools services in the virtual machine:
Switching to guest configuration: [ OK ]
Guest filesystem driver: [ OK ]
Guest vmxnet fast network device: [ OK ]
DMA setup: [ OK ]
Guest operating system daemon: [ OK ]
The configuration of VMware Tools e.x.p build-22874 for Linux
for this running
kernel completed successfully.
You must restart your X session before any mouse or graphics
changes take
effect.
You can now run VMware Tools by invoking the following
command:
"/usr/bin/vmware-toolbox" during an XFree86 session.
To use the vmxnet driver, restart networking using the following
commands:
/etc/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/network start
Enjoy,
--the VMware team
#
The VMware client tools are now installed.
Create Shared Disks
Shut down the RAC1 virtual machine using the following
command.
# shutdown -h now
Create a directory on the host system to hold the shared virtual
disks.
# mkdir -p /u01/VM/shared
On the VMware Server Console, click the "Edit virtual machine
settings" button. On the "Virtual Machine Settings" screen, click
the "+ Add" button.
Select the hardware type of "Hard Disk" and click the "Next"
button.
Accept the "Create a new virtual disk" option by clicking the
"Next" button.
Accept the "SCSI" option by clicking the "Next" button.
Set the disk size to "10.0" GB and uncheck the "Allocate all
disk space now" option, then click the "Next" button.
Set the disk name to "/u01/VM/shared/ocr.vmdk" and click the
"Advanced" button.
Set the virtual device node to "SCSI 1:1" and the mode to
"Independent" and "Persistent", then click the "Finish" button.
Repeat the previous hard disk creation steps 4 more times, using
the following values:
File Name: /u01/VM/shared/votingdisk.vmdkVirtual Device Node:
SCSI 1:2Mode: Independent and Persistent
File Name: /u01/VM/shared/asm1.vmdkVirtual Device Node: SCSI
1:3Mode: Independent and Persistent
File Name: /u01/VM/shared/asm2.vmdkVirtual Device Node: SCSI
1:4Mode: Independent and Persistent
File Name: /u01/VM/shared/asm3.vmdkVirtual Device Node: SCSI
1:5Mode: Independent and Persistent
At the end of this process, the virtual machine should look
something like the picture below.
Edit the contents of the "/u01/VM/RAC1/RAC1.vmx" file using a
text editor, making sure the following entries are present. Some of
the tries will already be present, some will not.
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.fileName = "/u01/VM/shared/ocr.vmdk"
scsi1:1.deviceType = "plainDisk"
scsi1:1.redo = ""
scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.fileName = "/u01/VM/shared/votingdisk.vmdk"
scsi1:2.deviceType = "plainDisk"
scsi1:2.redo = ""
scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.fileName = "/u01/VM/shared/asm1.vmdk"
scsi1:3.deviceType = "plainDisk"
scsi1:3.redo = ""
scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.fileName = "/u01/VM/shared/asm2.vmdk"
scsi1:4.deviceType = "plainDisk"
scsi1:4.redo = ""
scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.fileName = "/u01/VM/shared/asm3.vmdk"
scsi1:5.deviceType = "plainDisk"
scsi1:5.redo = ""
Start the RAC1 virtual machine by clicking the "Power on this
virtual machine" button on the VMware Server Console. When the
server has started, log in as the root user so you can partition
the disks. The current disks can be seen by issueing the following
commands.
# cd /dev
# ls sd*
sda sda1 sda2 sdb sdc sdd sde sdf
#
Use the "fdisk" command to partition the disks sdb to sdf. The
following output shows the expected fdisk output for the sdb
disk.
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun,
SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory
only,
until you decide to write them. After that, of course, the
previous
content won't be recoverable.
The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than
1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of
LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be
corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default
1305):
Using default value 1305
Command (m for help): p
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 1305 10482381 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
#
In each case, the sequence of answers is "n", "p", "1",
"Return", "Return", "p" and "w".
Once all the disks are partitioned, the results can be seen by
repeating the previous "ls" command.
# cd /dev
# ls sd*
sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1 sdf sdf1
#
Edit the /etc/sysconfig/rawdevices file, adding the following
lines.
/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1
/dev/raw/raw3 /dev/sdd1
/dev/raw/raw4 /dev/sde1
/dev/raw/raw5 /dev/sdf1
Restart the rawdevices service using the following command.
service rawdevices restart
Create some symbolic links to the raw devices. This is not
really necessary, but it acts as a reminder of the true
locatons.
ln -s /dev/raw/raw1 /u01/oradata/ocr
ln -s /dev/raw/raw2 /u01/oradata/votingdisk
ln -s /dev/raw/raw3 /u01/oradata/asm1
ln -s /dev/raw/raw4 /u01/oradata/asm2
ln -s /dev/raw/raw5 /u01/oradata/asm3
Run the following commands and add them the /etc/rc.local
file.
chown oracle:oinstall /dev/raw/raw1
chown oracle:oinstall /dev/raw/raw2
chown oracle:oinstall /dev/raw/raw3
chown oracle:oinstall /dev/raw/raw4
chown oracle:oinstall /dev/raw/raw5
chmod 600 /dev/raw/raw1
chmod 600 /dev/raw/raw2
chmod 600 /dev/raw/raw3
chmod 600 /dev/raw/raw4
chmod 600 /dev/raw/raw5
The shared disks are now configured.
Clone the Virtual Machine
The current version of VMware Server does not include an option
to clone a virtual machine, but the following steps illustrate how
this can be achieved manually.
Shut down the RAC1 virtual machine using the following
command.
# shutdown -h now
Copy the RAC1 virtual machine using the following command.
# cp -R /u01/VM/RAC1 /u01/VM/RAC2
Edit the contents of the "/u01/VM/RAC2/RAC1.vmx" file, making
the following change.
displayName = "RAC2"
Ignore discrepancies with the file names in the "/u01/VM/RAC2"
directory. This does not affect the action of the virtual
machine.
In the VMware Server Console, select the File > Open menu
options and browse for the "/u01/VM/RAC2/RAC1.vmx" file. Once
opened, the RAC2 virtual machine is visible on the console. Start
the RAC2 virtual machine by clicking the "Power on this virtual
machine" button and click the "Create" button on the subsequent
"Question" screen.
Ignore any errors during the server startup. We are expecting
the networking components to fail at this point.
Log in to the RAC2 virtual machine as the root user and start
the "Network Configuration" tool (Applications > System Settings
> Network).
Highlight the "eth0" interface and click the "Edit" button on
the toolbar and alter the IP address to "192.168.2.102" in the
resulting screen.
Click on the "Hardware Device" tab and click the "Probe" button.
Then accept the changes by clicking the "OK" button.
Repeat the process for the "eth1" interface, this time setting
the IP Address to "192.168.0.102".
Click on the "DNS" tab and change the host name to
"rac2.localdomain", then click on the "Devices" tab.
Once you are finished, save the changes (File > Save) and
activate the network interfaces by highlighting them and clicking
the "Activate" button. Once activated, the screen should look like
the following image.
Edit the /home/oracle/.bash_profile file on the RAC2 node to
correct the ORACLE_SID value.
ORACLE_SID=RAC2; export ORACLE_SID
Start the RAC1 virtual machine and restart the RAC2 virtual
machine. Whilst starting up, the "Kudzu" detection screen may be
displayed.
Press a key and accept the configuration change on the following
screen.
When both nodes have started, check they can both ping all the
public and private IP addresses using the following commands.
ping -c 3 rac1
ping -c 3 rac1-priv
ping -c 3 rac2
ping -c 3 rac2-priv
At this point the virtual IP addresses defined in the /etc/hosts
file will not work, so don't bother testing them.
It's a good idea to take a snapshot of the virtual machines, so
you can repeat the following stages if you run into any problems.
To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PreClusterware.tar RAC1 RAC2 shared
# gzip RAC-PreClusterware.tar
The virtual machine setup is now complete.
Install the Clusterware Software
Start the RAC1 and RAC2 virtual machines, login to RAC1 as the
oracle user and start the Oracle installer.
./runInstaller
On the "Welcome" screen, click the "Next" button.
Accept the default inventory location by clicking the "Next"
button.
Enter the appropriate name and path for the Oracle Home and
click the "Next" button.
Wait while the prerequisite checks are done. If you have any
failures correct them and retry the tests before clicking the
"Next" button.
You can choose to ignore the warnings from the prerequisite
checks and click the "Next" button. If you do, you will also need
to ignore the subsequent warning message by clicking the "Yes"
button.
The "Specify Cluster Configuration" screen shows only the RAC1
node in the cluster. Click the "Add" button to continue.
Enter the details for the RAC2 node and click the "OK"
button.
Click the "Next" button to continue.
The "Specific Network Interface Usage" screen defines how each
network interface will be used. Highlight the "eth0" interface and
click the "Edit" button.
Set the "eht0" interface type to "Public" and click the "OK"
button.
Leave the "eth1" interface as private and click the "Next"
button.
Click the "External Redundancy" option, enter "/dev/raw/raw1" as
the OCR Location and click the "Next" button. To have greater
redundancy we would need to define another shared disk for an
alternate location.
Click the "External Redundancy" option, enter "/dev/raw/raw2" as
the Voting Disk Location and click the "Next" button. To have
greater redundancy we would need to define another shared disk for
an alternate location.
On the "Summary" screen, click the "Install" button to
continue.
Wait while the installation takes place.
Once the install is complete, run the orainstRoot.sh and root.sh
scripts on both nodes as directed on the following screen.
The output from the orainstRoot.sh file should look something
like that listed below.
# cd /u01/app/oracle/oraInventory
# ./orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to
oinstall.
The execution of the script is complete
#
The output of the root.sh will vary a little depending on the
node it is run on. The following text is the output from the RAC1
node.
# cd /u01/crs/oracle/product/10.2.0/crs
# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned
by root
WARNING: directory '/u01/crs/oracle/product' is not owned by
root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned
by root
WARNING: directory '/u01/crs/oracle/product' is not owned by
root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw2
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
CSS is inactive on these nodes.
rac2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
#
Ignore the directory ownership warnings. We should really use a
separate directory structure for the clusterware so it can be owned
by the root user, but it has little effect on the finished
results.
The output from the RAC2 node is listed below.
# cd /u01/crs/oracle/product/10.2.0/crs
# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned
by root
WARNING: directory '/u01/crs/oracle/product' is not owned by
root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned
by root
WARNING: directory '/u01/crs/oracle/product' is not owned by
root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been
initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
rac2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces
should be used to configure virtual IPs.
#
Here you can see that some of the configuration steps are
omitted as they were done by the first node. In addition, the final
part of the script ran the Virtual IP Configuration Assistant
(VIPCA) in silent mode, but it failed. This is because my public IP
addresses are actually within the "192.168.255.255" range which is
a private IP range. If you were using "legal" IP addresses you
would not see this and you could ignore the following VIPCA
steps.
Run the VIPCA manually as the root user on the RAC2 node using
the following command.
# cd /u01/crs/oracle/product/10.2.0/crs/bin
# ./vipca
Click the "Next" button on the VIPCA welcome screen.
Highlight the "eth0" interface and click the "Next" button.
Enter the vitual IP alias and address for each node. Once you
enter the first alias, the remaining values should default
automatically. Click the "Next" button to continue.
Accept the summary information by clicking the "Finish"
button.
Wait until the configuration is complete, then click the "OK"
button.
Accept the VIPCA results by clicking the "Exit" button.
You should now return to the "Execute Configuration Scripts"
screen on RAC1 and click the "OK" button.
Wait for the configuration assistants to complete.
When the installation is complete, click the "Exit" button to
leave the installer.
It's a good idea to take a snapshot of the virtual machines, so
you can repeat the following stages if you run into any problems.
To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PostClusterware.tar RAC1 RAC2 shared
# gzip RAC-PostClusterware.tar
The clusterware installation is now complete.
Install the Database Software and Create an ASM Instance
Start the RAC1 and RAC2 virtual machines, login to RAC1 as the
oracle user and start the Oracle installer.
./runInstaller
On the "Welcome" screen, click the "Next" button.
Select the "Enterprise Edition" option and click the "Next"
button.
Enter the name and path for the Oracle Home and click the "Next"
button.
Select the "Cluster Install" option and make sure both RAC nodes
are selected, the click the "Next" button.
Wait while the prerequisite checks are done. If you have any
failures correct them and retry the tests before clicking the
"Next" button.
You can choose to ignore the warnings from the prerequisite
checks and click the "Next" button. If you do, you will also need
to ignore the subsequent warning message by clicking the "Yes"
button.
Select the "Configure Automatic Storage Management (ASM)"
option, enter the SYS password for the ASM instance, then click the
"Next" button.
Select the "External" redundancy option (no mirroring), select
all three raw disks (raw3, raw4 and raw5), then click the "Next"
button.
On the "Summary" screen, click the "Install" button to
continue.
Wait while the database software installs.
Once the installation is complete, wait while the configuration
assistants run.
Execute the "root.sh" scripts on both nodes, as instructed on
the "Execute Configuration scripts" screen, then click the "OK"
button.
When the installation is complete, click the "Exit" button to
leave the installer.
It's a good idea to take a snapshot of the virtual machines, so
you can repeat the following stages if you run into any problems.
To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PostASM.tar RAC1 RAC2 shared
# gzip RAC-PostASM.tar
The database software installation and ASM creation step is now
complete.
Create a Database using the DBCA
Start the RAC1 and RAC2 virtual machines, login to RAC1 as the
oracle user and start the Database Configuration Assistant.
dbca
On the "Welcome" screen, select the "Oracle Real Application
Clusters database" option and click the "Next" button.
Select the "Create a Database" option and click the "Next"
button.
Highlight both RAC nodes and click the "Next" button.
Select the "Custom Database" option and click the "Next"
button.
Enter the values "RAC.WORLD" and "RAC" for the Global Database
Name and SID Prefix respectively, then click the "Next" button.
Accept the management options by clicking the "Next" button. If
you are attempting the installation on a server with limited
memory, you may prefer not to configure Enterprise Manager at this
time.
Enter database passwords then click the "Next" button.
Select the "Automatic Storage Management (ASM)" option, then
click the "Next" button.
Select the "DATA" disk group, then click the "Next" button.
Accept the "Use Oracle-Managed Files" database location by the
"Next" button.
Check both the "Specify Flash Recovery Area" and "Enable
Archiving" options. Enter "+DATA" as the Flash Recovery Area, then
click the "Next" button.
Uncheck all but the "Enterprise Manager Repository" option, then
click the "Standard Database Components..." button.
Uncheck all but the "Oracle JVM" option, then click the "OK"
button, followed by the "Next" button on the previous screen. If
you are attempting the installation on a server with limited
memory, you may prefer not to install the JVM at this time.
Accept the current database services configuration by clicking
the "Next" button.
Select the "Custom" memory management option and accept the
default settings by clicking the "Next" button.
Accept the database storage settings by clicking the "Next"
button.
Accept the database creation options by clicking the "Finish"
button.
Accept the summary information by clicking the "OK" button.
Wait while the database is created.
Once the database creation is complete you are presented with
the following screen. Make a note of the information on the screen
and click the "Exit" button.
The RAC database creation is now complete.
TNS Configuration
Once the installation is complete, the
"$ORACLE_HOME/network/admin/listener.ora" file on each RAC node
will contain entries similar to the following.
LISTENER_RAC1 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.101)(PORT =
1521)(IP = FIRST))
)
)
SID_LIST_LISTENER_RAC1 =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
(PROGRAM = extproc)
)
)
The "$ORACLE_HOME/network/admin/tnsnames.ora" file on each RAC
node will contain entries similar to the following.
RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
)
)
LISTENERS_RAC =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
)
RAC2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
(INSTANCE_NAME = RAC2)
)
)
RAC1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
(INSTANCE_NAME = RAC1)
)
)
This configuration allows direct connections to specific
instance, or using a load balanced connection to the main
service.
$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Apr 18 12:27:11
2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, OLAP and Data
Mining options
SQL> CONN sys/password@rac1 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME HOST_NAME
----------------
----------------------------------------------------------------
RAC1 rac1.localdomain
SQL> CONN sys/password@rac2 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME HOST_NAME
----------------
----------------------------------------------------------------
RAC2 rac2.localdomain
SQL> CONN sys/password@rac AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME HOST_NAME
----------------
----------------------------------------------------------------
RAC1 rac1.localdomain
SQL>
Check the Status of the RAC
There are several ways to check the status of the RAC. The
srvctl utility shows the current configuration and status of the
RAC database.
$ srvctl config database -d RAC
rac1 RAC1 /u01/app/oracle/product/10.2.0/db_1
rac2 RAC2 /u01/app/oracle/product/10.2.0/db_1
$
$ srvctl status database -d RAC
Instance RAC1 is running on node rac1
Instance RAC2 is running on node rac2
$
The V$ACTIVE_INSTANCES view can also display the current status
of the instances.
$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Apr 18 12:15:15
2006
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, OLAP and Data
Mining options
SQL> SELECT * FROM v$active_instances;
INST_NUMBER INST_NAME
-----------
------------------------------------------------------------
1 rac1.localdomain:RAC1
2 rac2.localdomain:RAC2
SQL>
Finally, the GV$ allow you to display global information for the
whole RAC.
SQL> SELECT inst_id, username, sid, serial# FROM gv$session
WHERE username IS NOT NULL;
INST_ID USERNAME SID SERIAL#
---------- ------------------------------ ----------
----------
1 SYS 127 2
1 SYS 128 28
1 SYS 130 10
1 SYS 131 4
1 SYS 133 9
1 DBSNMP 134 27
1 DBSNMP 135 1
1 SYS 153 122
2 SYSMAN 120 243
2 DBSNMP 122 37
2 DBSNMP 124 93
INST_ID USERNAME SID SERIAL#
---------- ------------------------------ ----------
----------
2 SYSMAN 125 2
2 SYSMAN 127 6
2 SYS 128 26
2 SYS 129 30
2 SYS 130 3
2 SYS 133 149
2 SYSMAN 134 58
2 SYS 136 32
19 rows selected.
SQL>
If you have configured Enterprise Manager, it can be used to
view the configuration and current status of the database.
For more information see:
Oracle Database 10g Release 2 (10.2.0.1) RAC Installation On
Windows 2003 Using VMware Server
Oracle 11g Release 1 RAC On Linux Using VMware Server
Oracle 11g Release 1 RAC On Linux Using NFS
Build Your Own Oracle RAC 10g Release 2 Cluster on Linux and
FireWire
RACing ahead with Oracle on VMware Series (Start of series)
Installation Guide for Linux x86 (10.2)
Oracle Clusterware and Oracle Real Application Clusters
Installation Guide for Linux (10.2)
Configuring I/O for Raw Partitions
Hope this helps. Regards Tim...