Build Oracle RAC 11.2.0.3 on Solaris 11 Using Oracle Virtual Box

Build Oracle RAC 11.2.0.3 on Oracle Solaris 11 11.11 using

Oracle VirtualBox

In the article you will have a look at how to use some Oracle VirtualBox

features to build two node Oracle 11gR2 11.2.0.3 RAC system on Oracle Solaris

11 11.11 x86-64. For information about a similar two node Oracle RAC 11.2

setup on Solaris 10 x86-64 using VirtualBox click here. The article will

emphasize on the Solaris 11 management options and configurations required to

meet Oracle 11.2.0.3 RAC installation prerequisites.

The following software will be used:

1. Oracle 11.2.0.3 for Solaris (x86-64)- patch 10404530. Download from MOS

here.

2. Oracle Solaris 11 (x86-64). I used Oracle Solaris 11 11/11 Live Media

for x86. Download from here.

3. Oracle VM VirtualBox 4.1.8 Download from here.

There will be two virtual machines Sol1 and Sol2, each of them will be

configured with

4GB RAM

160GB bootable disk

NIC – NAT: for access to Internet

NIC - bridged for public interface in RAC with address

192.168.2.21/22 (first IP 192.168.2.21 on sol1 and second IP

192.168.2.22 on node sol2). These are public interface in RAC.

NIC – bridged for private interface in RAC with address

10.10.2.21/22 (first IP 10.10.2.21 on sol1 and second IP

10.10.2.22 on node sol2). These are private interface in RAC.

NIC – Host Only : for FTP/SSH/telnet/SCP access from the host OS

to the guest OS. (first IP 192.168.56.51 on sol1 and second IP

192.168.56.51on node sol2).

5 10GB attached shared disks for the ASM storage.

Sol1 VM will run Solaris 11 guest with hostname sol1. Sol2 VM

will run Solaris 11 guest with hostname sol2.

Note. For access to the Guest OS (Solaris) from the Host (MS Windows Vista

64) via ssh/scp/telnet/ftp.

For the host OS to be able to access the guest OS a Host Only adapter is

required and the corresponding IP on the guest OS should be within the IP

subnet of the VirtualBox Host-Only adapter. On the host (Windows in this

case) we have.

Ethernet adapter VirtualBox Host-Only Network:

Connection-specific DNS Suffix . :

Description . . . . . . . . . . . : VirtualBox Host-Only Ethernet Adapter

Physical Address. . . . . . . . . : 08-00-27-00-3C-B7

DHCP Enabled. . . . . . . . . . . : No

http://gjilevski.wordpress.com/2011/05/10/virtualization-using-oracle-vm-virtualbox-for-building-two-node-oracle-rac-11gr2-cluster-based-on-solaris-10-x86-64/

support.oracle.com

http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html

Autoconfiguration Enabled . . . . : Yes

IPv4 Address. . . . . . . . . . . : 192.168.56.1(Preferred)

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . :

NetBIOS over Tcpip. . . . . . . . : Enabled

Thus, the IP, corresponding to the Host-Only NIC, inside the guest (Solaris

in this case) should be within the 192.168.56.* subnet. Should the VirtualBox

Host-Only adapter IP change, make sure that the IP within the guest OS is

within the same subnet as the IP of the VirtualBox Host-Only adapter.

Examples of accessing Solaris 11 from a cygwin on Windows using ssh/scp.

bash-3.2$ uname -a

CYGWIN_NT-6.0-WOW64 userpc 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin

bash-3.2$

bash-3.2$ ssh -X [email protected]

The authenticity of host '192.168.56.52 (192.168.56.52)' can't be established.

RSA key fingerprint is 1b:f8:c2:74:cf:29:4b:e8:0d:a6:d8:f6:d9:51:92:72.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.56.52' (RSA) to the list of known hosts.

Password:

Warning: untrusted X11 forwarding setup failed: xauth key data not generated

Warning: No xauth data; using fake authentication data for X11 forwarding.

Last login: Mon Feb 6 22:22:49 2012

Oracle Corporation SunOS 5.11 11.0 November 2011

gjilevski@sol2:~$ uname -a

SunOS sol2 5.11 11.0 i86pc i386 i86pc

gjilevski@sol2:~$ ls /u01/sh.sh

/u01/sh.sh

gjilevski@sol2:~$ exit

logout

Connection to 192.168.56.52 closed.

bash-3.2$ scp [email protected]:/u01/sh.sh .

Password:

sh.sh 100% 682 0.7KB/s 00:00

bash-3.2$ uname -a

CYGWIN_NT-6.0-WOW64 userpc 1.7.5(0.225/5/3) 2010-04-12 19:07 i686 Cygwin

bash-3.2$

Same approach can be used for telnet or ftp access from host OS if ftp or

telnet services are enabled on the guest Solaris OS.

First you will create the first virtual machine and configure network

interfaces and install Solaris software and meet the Oracle RAC installation

prerequisites. You will than attach the shared disks to the first virtual

machine and prepare the shared storage for ASM. Last you will clone the boot

disk of the first VM to create the second VM and plug the shared storage and

change the IPs and hostname. In this article you will have role separation

with two different accounts one (grid) for ASM storage management and the

second (oracle) for RDBMS installation. In order to support role separation

you will create additional groups for (asmadmin, asmdba, asmoper, oinstall)

and ( dba. oinstall, oper). At the end you will install GI, RDBMS and create

a two node RAC database. You will see how to fix the problems that were

encountered during the installation.

1. Create the first VM Sol1

Oracle Solaris 11 was released late 2011. In order to obtain detailed

information for installation options click here or access Solaris 11

documentation click here.

I am using Oracle Solaris 11 11/11 Live Media for x86. It performs

default installation using Automatic network management. After Solaris

11 installation, I will switch to Manual network management and modify

IP addresses and hostnames.

Select New and click Next. Enter the name of the VM (Sol1) and press

Next.

Select 4096 MB for the RAM of the VM and press Next to continue.

http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-112-s11-first-steps-524819.html

http://docs.oracle.com/cd/E23824_01/index.html

Select create a new disk for the boot disk and press Next to continue.

Select Dynamically expanding storage and press Next to continue.

Select 160GB (not 16GB) and press Next to continue. (16GB is not sufficient

to install Solaris and Oracle GI and RDBMS).

Press Next to continue.

Press Finish.

This concludes the VM creation. After that select the VM click settings and

add the four NIC as specified. First to be NAT, bridged, bridged and Host-

Only.

Create the disks to be used as shared.

VBoxManage createhd --filename d:\vb\asm1.vdi --size 10240 --format VDI --variant Fixed










Attach the shared disks to the VM and mark them as shared. Note that for a

disk to be shared it must be fixed.

VBoxManage storageattach Sol1 --storagectl "SATA Controller" --port 1 --device 0 -

-type hdd --medium d:\vb\asm1.vdi --mtype shareable

















VBoxManage storageattach Sol1 --storagectl "SATA Controller" --port 10 --device 0

--type hdd --medium d:\vb\asm10.vdi --mtype shareable

VBoxManage modifyhd d:\vb\asm1.vdi --type shareable










The configuration will look like the graphical image below.

Now that the shared disks are attached and the virtual NICs are specified

let’s start the VM for Solaris installation. After the VM is created and

started specify the media for Solaris installation. The iso image file can be

either staged in a directory on the host OS or on DVD. Start the VM and

specify the installation media containing the Solaris iso. In this case the

iso media is on a Windows directory and you press on the yellow folder with

green arrow to select it and press Next to continue.

Note that although the image is for Solaris 10 but you must use Solaris 11

iso file and will work the same way.

Press Next to continue.

Press Finish to start. Once the system boots up Select Solaris 11 and press

Return.

Select keyboard.

Select language.

Wait until the virtual OS is launched in memory. Note that Automatic Network

is selected. In order to start the real installation click Install Oracle

Solaris.

Press Next.

Select the 160GB disk. I did not partition the disk but later added in swap

space for the OUI to succeed. The default swap is 1GB and OUI asks for

minimum 4GB in order to install 11.2.0.3 RAC.

Select your time zone.

Create an account with administrative privileges. Starting with Solaris 11

root is defined as a role and you need a user to log in. A role cannot log

you in. Specify the hostname.

Review and press Install.

Wait for the install to completes.

Reboot once installation completes.

Select Solaris 11.

Login in with the previously created user.

Select login session.

Finally you are complete the Solaris 11 installation.

2. Installing Guest Additions

After login as yourself, on the VirtualBox console select Devices->Install

Guest Additions… The Virtual Editions media gets mounted as a virtual CD.

Select OK.

Here you have a chance to change the root password. Initially, use the

default (solaris) which is expired and need a change.

Reboot the Solaris guest VM.

3. Shared Folders configuration

Shared folders are configured as follows.

VirtualBox allows you to mount on the guest OS folders from the host OS.

If using latest VirtualBox 4.1 and later you can automatically add shared

folders and VirtualBox mounts them for you.

In case you are on a previous VirtualBox release, as root user create

directories for each folder and use the following command to mount.

pfexec mount -F vboxfs Solaris10 /whatever-you-want

In my case I created two directories /software and /OracleVMServer and

mounted the shared folders as follow.

pfexec mount -F vboxfs software /software

pfexec mount -F vboxfs OracleVMServer /OracleVMServer

To create a permanent mount points that persist a reboot add the following

line to the /etc/vfstab file.

software - /software vboxfs - yes -

OracleVMServer - /OracleVMServer vboxfs - yes -

4. Setting the shared storage for ASM and SWAP

The default configured swap space was 1GB. For Oracle 11.2 installation 4GB

minimum is required. Using the following commands drop and recreate the swap

with desired size 5GB.

root@sol1:~# swap -d /dev/zvol/dsk/rpool/swap

root@sol1:~# zfs volsize=5G rpool/swap

root@sol1:~# swap -a /dev/zvol/dsk/rpool/swap

root@sol1:~# swap -s

total: 259856k bytes allocated + 35808k reserved = 295664k used,

6669044k available

root@sol1:~# swap -l

swapfile dev swaplo blocks free

/dev/zvol/dsk/rpool/swap 195,2 8 10485752 10485752

root@sol1:~#

There are ten shared disks attached to the Solaris VM. Those disks are

visible as c1t1d0, c1t2d0,c1t3d0,c1t4d0, c1t5d0, c1t6d0, c1t7d0, c1t8d0,

c1t9d0 and c1t10d0. You need to format and prepare them for the ASM with

privileges (0660) and ownership (grid:asmadmin). Each disk is formatted and a

partition is created starting from 1 cylinder. The available disks can be

seen using the format command as shown below. Than for each disk you invoke

fdisk and accept Solaris as default type of partition. Then you partition

with partition command and select 0 partition to start from first cylinder

and enter size 9.95GB (remember 10GB was the original size) and print the

partition table. You repeat the procedure for all ten shared disks. The flow

of commands is as follow.

root@sol1:~# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c1t0d0 <DEFAULT cyl 20883 alt 2 hd 255 sec 63>

/pci@0,0/pci8086,2829@d/disk@0,0


/pci@0,0/pci8086,2829@d/disk@1,0


/pci@0,0/pci8086,2829@d/disk@2,0


/pci@0,0/pci8086,2829@d/disk@3,0


/pci@0,0/pci8086,2829@d/disk@4,0


/pci@0,0/pci8086,2829@d/disk@5,0


/pci@0,0/pci8086,2829@d/disk@6,0


/pci@0,0/pci8086,2829@d/disk@7,0


/pci@0,0/pci8086,2829@d/disk@8,0


/pci@0,0/pci8086,2829@d/disk@9,0


/pci@0,0/pci8086,2829@d/disk@a,0

Specify disk (enter its number): 4

selecting c1t4d0

[disk formatted]

FORMAT MENU:

disk - select a disk

type - select (define) a disk type

partition - select (define) a partition table

current - describe the current disk

format - format and analyze the disk

fdisk - run the fdisk program

repair - repair a defective sector

label - write label to the disk

analyze - surface analysis

defect - defect list management

backup - search for backup labels

verify - read and display labels

save - save new disk/partition definitions

inquiry - show vendor, product and revision

volname - set 8-character volume name

!<cmd> - execute <cmd>, then return

quit

format> format>

format> fdisk

No fdisk table exists. The default partition for the disk is:

a 100% "SOLARIS System" partition

Type "y" to accept the default partition, otherwise type "n" to edit the

partition table.

y

format>

format> partition

PARTITION MENU:

0 - change `0' partition








select - select a predefined table

modify - modify a predefined partition table

name - name the current table

print - display the current table

label - write partition map and label to the disk

!<cmd> - execute <cmd>, then return

quit

partition> 0

Part Tag Flag Cylinders Size Blocks

0 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]:

Enter partition permission flags[wm]:

Enter new starting cyl[0]: 1

Enter partition size[0b, 0c, 1e, 0.00mb, 0.00gb]: 9.95g

partition> print

Current partition table (unnamed):

Total disk cylinders available: 1302 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks

0 unassigned wm 1 - 1299 9.95GB (1299/0/0) 20868435


2 backup wu 0 - 1301 9.97GB (1302/0/0) 20916630






8 boot wu 0 - 0 7.84MB (1/0/0) 16065


partition>

partition> label

Ready to label disk, continue? y

partition>quit

In this case the whole disk c1t4d0 was partitioned as one partition and made

available for ASM. Repeat the same steps for all remaining nine shared disks.

Since slice 0 was used you need to make those available to ASM by setting the

permissions and ownership as show below.

chown grid:asmadmin /dev/rdsk/c1t1d0s0

chmod 660 /dev/rdsk/c1t1d0s0



















For more information refer to Oracle documentation here.

5. Creating Solaris users and groups

Oracle starting with 11gR2 is enabling a role separation when installing

Oracle GI and Oracle RDBMS. The idea is Oracle GI will have a user grid as

ASM administrator and Oracle RDBMS will have a user oracle as RDBMS

administrator. Both users share oinstall group so that both Oracle GI and

Oracle RDBMS share the same inventory. However each admin has different

groups. The grid user is a member of asmadmin, asmdba, asmoper and the

primary group is oinstall. The oracle user is a member of dba, asmdba and

oper and the primary group is oinstall. Users and groups are created with

the following commands.

groupadd -g 1000 oinstall

groupadd -g 1020 asmadmin

groupadd -g 1021 asmdba

groupadd -g 1022 asmoper

groupadd -g 1031 dba

groupadd -g 1032 oper

useradd -u 1100 -g oinstall -G asmoper,asmadmin,asmdba -d /export/home/grid -m grid

useradd -u 1101 -g oinstall -G oper,dba,asmdba -d /export/home/oracle -m oracle

http://download.oracle.com/docs/cd/E11882_01/install.112/e17213/storage.htm#CDECEBGH

http://download.oracle.com/docs/cd/E11882_01/install.112/e17213/presolar.htm#CHDBBHGF

User grid and user oracle separate the responsibilities between storage

administrator and Oracle DBA.

6. Create the directories for ORACLE_BASE and ORACLE_HOME for

Oracle GI and Oracle RDBMS

mkdir -p /u01/app/11.2.0/grid

mkdir -p /u01/app/grid

chown -R grid:oinstall /u01

mkdir -p /u01/app/oracle

chown oracle:oinstall /u01/app/oracle

chmod -R 775 /u01

mkdir -p /u01/app/11.2.0/grid

chown grid:oinstall /u01/app/11.2.0/grid

chmod -R 775 /u01/app/11.2.0/grid

mkdir -p /u01/app/oracle/product/11.2.0/db_1

chown -R oracle:oinstall /u01/app/oracle

chmod -R 775 /u01/app/oracle

7. Setup Solaris kernel parameters.

Create two projects for Oracle GI and Oracle RDBMS users respectively. Set

the share memory parameters.

projadd -U grid -K "project.max-shm-memory=(priv,6g,deny)" user.grid

projmod -sK "project.max-sem-nsems=(priv,512,deny)" user.grid

projmod -sK "project.max-sem-ids=(priv,128,deny)" user.grid

projmod -sK "project.max-shm-ids=(priv,128,deny)" user.grid

projmod -sK "project.max-shm-memory=(priv,6g,deny)" user.grid

projadd -U oracle -K "project.max-shm-memory=(priv,6g,deny)" user.oracle

projmod -sK "project.max-sem-nsems=(priv,512,deny)" user.oracle

projmod -sK "project.max-sem-ids=(priv,128,deny)" user.oracle

projmod -sK "project.max-shm-ids=(priv,128,deny)" user.oracle

projmod -sK "project.max-shm-memory=(priv,6g,deny)" user.oracle

/usr/sbin/projmod -sK "process.max-file-descriptor=(priv,65536,deny)" user.oracle

/usr/sbin/projmod -sK "process.max-file-descriptor=(priv,65536,deny)" user.grid

If the max file descriptors are not set properly there might be errors

starting OUI.

Set the TCP and UDP kernel parameters

/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000

/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500

/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000

/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500

Put in /etc/inittab the following lines for the TCP and UDP parameters to

persist across reboot.

tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_smallest_anon_port 9000 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/tcp tcp_largest_anon_port 65500 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_smallest_anon_port 9000 > /dev/console

tm::sysinit:/usr/sbin/ndd -set /dev/udp udp_largest_anon_port 65500 > /dev/console

8. Automatic SSH configuration

Oracle recommends using OUI to setup ssh user equivalence where users can

connect across the node of the cluster without a password. To avoid errors

while attaching $OH when the remote node closes a connection prematurely set

in /etc/ssh/sshd_config file LoginGraceTime 0. After changing the file

restart ssh service.

svcadm restart ssh

9. Enable Core file creation

Make sure that Core Dumps are enabled. To check if core dumps are enabled use

coreadm command as follows.

# coreadm

global core file pattern:

global core file content: default

init core file pattern: core

init core file content: default

global core dumps: disabled

per-process core dumps: enabled

global setid core dumps: disabled

per-process setid core dumps: disabled

global core dump logging: disabled

#

As root user make the following directory. And enable core dumps as show

below.

mkdir -p /var/cores

coreadm -g /var/cores/%f.%n.%p.%t.core -e global -e global-setid -e

log -d process -d proc-setid

Verify the result.

# coreadm

global core file pattern: /var/cores/%f.%n.%p.%t.core

global core file content: default

init core file pattern: core

init core file content: default

global core dumps: enabled

per-process core dumps: disabled

global setid core dumps: enabled

per-process setid core dumps: disabled

global core dump logging: enabled

#

10. Network Time protocol Settings

You have two options for time synchronization: an operating system configured

network time protocol (NTP), or Oracle Cluster Time Synchronization Service.

Oracle Cluster Time Synchronization Service is designed for organizations

whose cluster servers are unable to access NTP services. If you use NTP, then

the Oracle Cluster Time Synchronization daemon (ctssd) starts up in observer

mode. If you do not have NTP daemons, then ctssd starts up in active mode and

synchronizes time among cluster members without contacting an external time

server. So there are two options

A. Disable NTP and rely entirely on CTSS:

As root execute the command below:

/usr/sbin/svcadm disable ntp

B. Configure NTP and use it together with CTSS:

As root edit the /etc/inet/ntp.conf file to add "slewalways yes" and "disable

pll" to the file. After you make these changes, restart xntpd using the

command

/usr/sbin/svcadm restart ntp.

NTP is disabled in the installation reflected in article.

11. Create the following profiles

For user grid

umask 022

ORACLE_BASE=/u01/app/grid

ORACLE_HOME=/u01/app/11.2.0/grid

ORACLE_SID=+ASM1

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_H OME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s unlimited

ulimit -v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

For user oracle

umask 022

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

ORACLE_SID=D11G

LD_LIBRARY_PATH=$ORACLE_HOME/lib

PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin

export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH

TEMP=/tmp

TMPDIR=/tmp

export TEMP TMPDIR

ulimit -t unlimited

ulimit -f unlimited

ulimit -d unlimited

ulimit -s unlimited

ulimit -v unlimited

if [ -t 0 ]; then

stty intr ^C

fi

12. Set up file descriptors

Edit the /etc/system to add the following parameters.

# Hard limit on file descriptors for single process

set rlim_fd_max = 65536

# Soft limit on the file descriptors for a single process

set rlim_fd_cur = 65536

13. Require packages and patches

For Solaris 10 x86-64 the following packages and patches are required.

SUNWarc

SUNWbtool

SUNWcsl

SUNWhea

SUNWlibC

SUNWlibm

SUNWlibms

SUNWsprot

SUNWtoo

SUNWi1of (ISO8859-1)

SUNWi1cs (ISO8859-15)

SUNWi15cs

SUNWxwfnt

119961-05 or later

119964-14 or later

120754-06 or later

139556-08 or later

139575-03 or later

141415-04 or later

141445-09 or later [11.2.0.2]

Since I am on Solaris 11 I did

root@sol2:~# pkg install SUNWarc SUNWbtool SUNWhea SUNWlibC SUNWlibm SUNWlibms SUNW

SUNWtoo SUNWi1of SUNWi1cs SUNWi15cs SUNWxwfnt SUNWcsl

Creating Plan

pkg install: The following pattern(s) did not match any allowable packages. Try

using a different matching pattern, or refreshing publisher information:

SUNWi1of

SUNW

SUNWxwfnt

root@sol2:~#

Solaris 11 is supported and certified for Oracle 11.2.0.3. Despite, that I

did not find information about packages and patches for Oracle 11.2.0.3 OUI

confirmed later that installed packages and patches suffice for Oracle

11.2.0.3 RAC installation on Solaris 11.

14. Configure IP addresses and confirm hostname

The performed Oracle Solaris 11 11/11 Live Media installation provides

Automatic Network Management. In order to manually set up IP addresses

I switched to Manual Network management using the following command.

root@sol1:~# netadm enable -p ncp DefaultFixed

Enabling ncp 'DefaultFixed'

root@sol1:~#

After switch to manual Network Management I have.

gjilevski@sol1:~$ dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE

net1 Ethernet unknown 0 unknown e1000g1




gjilevski@sol1:~$

In Solaris 11 IP addresses are set as follows:

ipadm create-ip net0

ipadm create-addr -T dhcp net0/addr


ipadm create-addr -T static -a local=192.168.2.21/24 net1/addr





At the end I have:

root@sol1:~# dladm show-phys

LINK MEDIA STATE SPEED DUPLEX DEVICE

net1 Ethernet up 1000 full e1000g1




root@sol1:~#

Hostname was configured during Solaris 11 installation.

15. Modify /etc/hosts file to have entries for the public,

private and scan addresses similarly to the output below

gjilevski@sol2:~$ cat /etc/hosts # # Copyright 2009 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # # Internet host table # ::1 localhost 127.0.0.1 localhost 192.168.2.21 sol1 sol1.gj.com 192.168.2.22 sol2 sol2.gj.com 192.168.2.31 sol1-vip sol1-vip.gj.com 192.168.2.32 sol2-vip sol2-vip.gj.com 192.168.2.51 scan-sol scan-sol.gj.com 10.10.2.21 sol1-priv sol1-priv.gj.com 10.10.2.22 sol2-priv sol2-priv.gj.com 192.168.56.51 sol1-exp sol1-exp.gj.com 192.168.56.52 sol2-exp sol2-exp.gj.com gjilevski@sol2:~$

16. Clone the boot disk of Sol1 VM to create a second VM

corresponding to the second node.

d:\VB>VBoxManage createhd --filename d:\vb\asm_5.vdi --size 10240 --format VDI -

-variant Fixed

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Disk image created. UUID: 0df463ae-587a-4f62-bcd1-ac7dbab86f8f

d:\VB>

Create a second VM called Sol2 with the same resource as Sol1 but specify the

cloned disk as a bootable disk. Attach the shared disks to the second VM

using commands as described below.

VBoxManage storageattach Sol2 --storagectl "SATA Controller" --

port 1 --device 0 --type hdd --medium d:\vb\asm1.vdi --mtype

shareable



shareable



shareable



shareable



shareable



shareable



shareable



shareable



shareable



shareable











17. Modify the hostname and IPs on the cloned VM Sol2

In order to avoid collision with IP start only the second cloned VM and

change the IP addresses and hostname. Initially both machines apparently have

same IPs and hostname. As root on Sol2 VM execute the following commands.

In order to modify the hostname to sol2 execute the following command to

change the host and delete the interfaces and reboot the node.

svccfg -s svc:/system/identity:node setprop config/nodename =sol2 svcadm refresh svc:/system/identity:node

svcadm restart svc:/system/identity:node ipadm delete-ip net0 ipadm delete-ip net1 ipadm delete-ip net2

ipadm delete-ip net3

After restart as root execute the following commands to setup the IP

addresses.

ipadm create-ip net0 ipadm create-addr -T dhcp net0/addr

ipadm create-ip net1 ipadm create-addr -T static -a local=192.168.2.22/24 net1/addr

ipadm create-ip net2 ipadm create-addr -T static -a local=10.10.2.22/24 net2/addr ipadm create-ip net3 ipadm create-addr -T static -a local=192.168.56.52/24 net3/addr

root@sol2:~# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net1 Ethernet up 1000 full e1000g1

net2 Ethernet up 1000 full e1000g2 net0 Ethernet up 1000 full e1000g0 net3 Ethernet up 1000 full e1000g3

root@sol2:~#

18. Reboot the cloned Sol2 and fire up the Sol1

19. Use OUI to setup ssh user equivalence for user grid

and oracle.

20. Run cluvfy to verify the prerequisites.

Use cluvfy from the stage directory to make sure that prerequisites are

met.

./runcluvfy.sh stage -post hwos -n sol1,sol2 –verbose

/runcluvfy.sh stage -pre crsinst -n sol1,sol2 -verbose

21. Start OUI for Oracle GI installation.

Oracle 11.2.0.3 on Solaris 11 requires that variable AWT_TOOLKIT be

exported.

export AWT_TOOLKIT=XToolkit

Failure to do so will produce the following error.

Preparing to launch O racle Universal Installer from /tmp/OraInstall2012-02-04_05-50-46AM. Please wait ...grid@sol1:/u01/stage/grid$ Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/OraInstall2012-02-04_05-50-46AM/jdk/jre/lib/amd64/motif21/libmawt.so: ld.so.1: java: fatal: libXm.so.4: open failed: No such file or

directory at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753)

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1649) at java.lang.Runtime.load0(Runtime.java:769) at java.lang.System.load(System.java:968) at java.lang.ClassLoader$NativeLibrary.load(Native Method)

at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1670) at java.lang.Runtime.loadLibrary0(Runtime.java:822) at java.lang.System.loadLibrary(System.java:993)

at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50) at java.security.AccessController.doPrivileged(Native Method)

at java.awt.Toolkit.loadLibraries(Toolkit.java:1509) at java.awt.Toolkit.<clinit>(Toolkit.java:1530) at com.jgoodies.looks.LookUtils.isLowResolution(LookUtils.java:484) at com.jgoodies.looks.LookUtils.<clinit>(LookUtils.java:249)

at com.jgoodies.looks.plastic.PlasticLookAndFeel.<clinit>(PlasticLookAndFeel.java:135) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at javax.swing.SwingUtilities.loadSystemClass(SwingUtilities.java:1779)

at javax.swing.UIManager.setLookAndFeel(UIManager.java:453) at oracle.install.commons.util.Application.startup(Application.java:780) at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:165)

at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:182) at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:348) at oracle.install.ivw.crs.driver.CRSInstaller.startup(CRSInstaller.java:98) at oracle.install.ivw.crs.driver.CRSInstaller.main(CRSInstaller.java:105)

After issuing xhost + from a root connected terminal session and setting a

DISPLAY=:0 and AWT_TOOLKIT=XToolkit in a session logged as grid run OUI.

Select Skip software update and press Next to continue.

Select Install and configure GI

Select typical installation.

Specify the SCAN name and enter data for the second node.

Make sure that interfaces are properly selected.

Select ASM as storage option.

Specify dusk group name, redundancy and disks. I selected HIGH redundancy.

Specify inventory location.

Examine the prerequisites check.

Review the summary and press Install.

Wait for the OUI to prompt for the scripts to be run as root.

root@sol1:/u01/app/11.2.0/grid# ./root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /var/opt/oracle/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file:

/u01/app/11.2.0/grid/crs/install/crsconfig_params

User ignored Prerequisites during installation

OLR initialization - successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert

Adding Clusterware entries to inittab

CRS-2672: Attempting to start 'ora.mdnsd' on 'sol1'

CRS-2676: Start of 'ora.mdnsd' on 'sol1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'sol1'

CRS-2676: Start of 'ora.gpnpd' on 'sol1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'sol1'

CRS-2672: Attempting to start 'ora.gipcd' on 'sol1'

CRS-2676: Start of 'ora.gipcd' on 'sol1' succeeded

CRS-2676: Start of 'ora.cssdmonitor' on 'sol1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'sol1'

CRS-2672: Attempting to start 'ora.diskmon' on 'sol1'

CRS-2676: Start of 'ora.diskmon' on 'sol1' succeeded

CRS-2676: Start of 'ora.cssd' on 'sol1' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 054f135ad3644fbabff3f058ef28ff75.

Successful addition of voting disk 834ba35a0f6c4fe0bf04700b9edc2bb2.

Successful addition of voting disk 3a99cb045bd94f5ebf038d4b4584228c.

Successful addition of voting disk 133e3427001c4f0dbfa957587866693a.

Successful addition of voting disk 90b3a7e49b004ff3bf58d1aa6eb91bbb.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 054f135ad3644fbabff3f058ef28ff75 (/dev/rdsk/c4t1d0s0) [DATA]

2. ONLINE 834ba35a0f6c4fe0bf04700b9edc2bb2 (/dev/rdsk/c4t2d0s0) [DATA]

3. ONLINE 3a99cb045bd94f5ebf038d4b4584228c (/dev/rdsk/c4t3d0s0) [DATA]

4. ONLINE 133e3427001c4f0dbfa957587866693a (/dev/rdsk/c4t4d0s0) [DATA]

5. ONLINE 90b3a7e49b004ff3bf58d1aa6eb91bbb (/dev/rdsk/c4t5d0s0) [DATA]

Located 5 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'sol1'

CRS-2676: Start of 'ora.asm' on 'sol1' succeeded

CRS-2672: Attempting to start 'ora.DATA.dg' on 'sol1'

CRS-2676: Start of 'ora.DATA.dg' on 'sol1' succeeded

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

root@sol1:/u01/app/11.2.0/grid#

root@sol2:/u01/app/11.2.0/grid# ./root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Creating /usr/local/bin directory...

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /var/opt/oracle/oratab file...

Entries will be added to the /var/opt/oracle/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file:

/u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

OLR initialization - successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active

CSS daemon on node sol1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the

cluster

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

root@sol2:/u01/app/11.2.0/grid#

At Final configuration step CLUVFY fails.

INFO: ERROR:

INFO: PRVG-1101 : SCAN name "scan-sol" failed to resolve

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for "scan-sol" (IP

address: 192.168.2.51) failed

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN

name "scan-sol"

As this is the only error in the log I ignored it. The error is due to not

properly defined SCAN in DNS.

Exit the OUI.

Verify that GI is properly installed and configured.

grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl check cluster -all

**************************************************************

sol1:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

sol2:

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

**************************************************************

grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.LISTENER.lsnr

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.asm

ONLINE ONLINE sol1 Started


ora.gsd

OFFLINE OFFLINE sol1


ora.net1.network

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.ons

ONLINE ONLINE sol1

ONLINE ONLINE sol2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE sol1

ora.cvu


ora.oc4j


ora.scan1.vip


ora.sol1.vip


ora.sol2.vip


grid@sol2:/u01/app/11.2.0/grid/log/sol2$ crsctl stat res -t -init

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.asm

1 ONLINE ONLINE sol2 Started

ora.cluster_interconnect.haip


ora.crf


ora.crsd


ora.cssd


ora.cssdmonitor


ora.ctssd

1 ONLINE ONLINE sol2 ACTIVE:0

ora.diskmon

1 OFFLINE OFFLINE

ora.evmd


ora.gipcd


ora.gpnpd


ora.mdnsd


grid@sol2:/u01/app/11.2.0/grid/log/sol2$

22. Oracle RDBMS installation only

Log in as oracle user. Set and export the following variables.

export AWT_TOOLKIT=XToolkit

export DISPLAY=:0.0

Run the installer. Ignore the E-mail notification and press Next to continue.

Select skip software updates.

Select install software only.

Select both nodes and Oracle RAC database.

Select languages.

Select EE.

Select location.

Select groups.

Examine the checks.

Separate checks confirmed that I can continue.

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.net1.network

NAME=ora.net1.network

TYPE=ora.network.type

TARGET=ONLINE , ONLINE

STATE=ONLINE on sol1, ONLINE on sol2

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.sol1.vip

NAME=ora.sol1.vip

TYPE=ora.cluster_vip_net1.type

TARGET=ONLINE

STATE=ONLINE on sol1

root@sol1:/u01/app/11.2.0/grid/bin# ./crsctl stat res ora.sol2.vip

NAME=ora.sol2.vip

TYPE=ora.cluster_vip_net1.type

TARGET=ONLINE

STATE=ONLINE on sol2

root@sol1:/u01/app/11.2.0/grid/bin#

Ignore examine Summary and run installation.

Run the scripts.

Exit the installer.

23. Create a cluster database using dbca

If you intend to have separate disk groups as FRA create it. In my case I

created the following disk groups.

While logged in as oracle user and properly set environment variables run

dbca. Select RAC database.

Select Create a database.

Select a template.

Select Admin managed and both nodes.

Select OEM option.

Specify password(s).

Error!

Look at:

Dbca Does Not Show ASM Diskgroup Information [ID 1286434.1]

ASM Diskgroup Can Not Be Shown When Creating Database With DBCA [ID

1269734.1]

Turned out permission issue. Add setuid and setgid bit for oracle binary

under $GRID_HOME/bin:

$ chmod 6755 $GRID_HOME/bin/oracle

The oracle binary for both GRID_HOME/bin and RDBMS ORACLE_HOME/bin should

have 6755 permission, eg: -rwsr-s—x

Select a Disk Group.

Select FRA and Archiving options.

Select sample schemas.

Select parameters.

Press Next.

Select Create.

Review the summary.

Modify the .profile to include ORACLE_UNQNAME and start dbconsole.

Modify passwords and take note of the OEM DC URL.

At the end database is up and running.

gjilevski@sol1:/u01/app/11.2.0/grid/bin$ ./crsctl stat res -t

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.DATADG.dg

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.LISTENER.lsnr

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.asm



ora.gsd



ora.net1.network

ONLINE ONLINE sol1

ONLINE ONLINE sol2

ora.ons

ONLINE ONLINE sol1

ONLINE ONLINE sol2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr


ora.cvu


ora.d11g.db

1 ONLINE ONLINE sol1 Open

2 ONLINE ONLINE sol2 Open

ora.oc4j


ora.scan1.vip


ora.sol1.vip


ora.sol2.vip


gjilevski@sol1:/u01/app/11.2.0/grid/bin$

oracle@sol2:~$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Feb 6 18:24:05 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME

----------- ------------------------------------------------------------

1 sol1.gj.com:D11G1

2 sol2.gj.com:D11G2

SQL>

SQL>

OEM status is as follows for the cluster database.

OEM status is as follows for the cluster.

Build Oracle RAC 11.2.0.3 on Solaris 11 Using Oracle Virtual Box

Documents

type targetonline

manual network

installation

core file

automatic

configuration

specific root

database configuration