Oracle 10g RAC Installation Using the Ocfs and Asmlib and SAN Over Openfiler

DBA Tips Archive for Oracle

Add a Node to an Existing Oracle RAC 10g R2 Cluster on Linux - (RHEL 4.5)by Jeff Hunter, Sr. Database Administrator

Contents

1. Overview2. Hardware and Costs3. Install the Linux Operating System4. Network Configuration5. Configure Network Security on the Openfiler Storage Server6. Configure the iSCSI Initiator7. Create "oracle" User and Directories8. Configure the Linux Server for Oracle9. Configure the "hangcheck-timer" Kernel Module

10. Configure RAC Nodes for Remote Access using SSH11. All Startup Commands for New Oracle RAC Node12. Install and Configure Oracle Cluster File System (OCFS2)13. Install and Configure Automatic Storage Management (ASMLib 2.0)14. Pre-Installation Tasks for Oracle10g Release 215. Extend Oracle Clusterware Software to the New Node16. Extend Oracle Database Software to the New Node17. Add Listener to New Node18. Add Database Instance to the New Node19. About the Author

Overview

As your organization grows so too does your need for more application and database resources to support the company'sIT systems. Oracle RAC 10g provides a scalable framework which allows DBA's to effortlessly extend the database tier tosupport this increased demand. As the number of users and transactions increase, additional Oracle instances can beadded to the Oracle database cluster to distribute the extra load.

This document is an extension to my article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 /iSCSI)". Contained in this new article are the steps required to add a single node to an already running and configuredtwo-node Oracle RAC 10g Release 2 environment on the CentOS 32-bit (x86) platform. Although this article was writtenand tested on CentOS 4.5 Linux, it should work unchanged with Red Hat Enterprise Linux 4 Update 5.

This article assumes the following:

The reader has already built and configured a two-node Oracle RAC 10g Release 2environment using the article "Building an Inexpensive Oracle RAC 10g Release 2 on Linux -(CentOS 4.5 / iSCSI)". The article provides comprehensive instructions for building a two-nodeRAC cluster, each with a single processor running CentOS 4.5, Oracle RAC 10g Release 2,OCFS2, and ASMLib 2.0. The current two-node RAC environment actually consists of threemachines two named linux1 and linux2 which each run an Oracle10g instance and a thirdnode to run the network storage server named openfiler1.

Note: The current two-node Oracle RAC environment has been upgraded from its base release(10.2.0.1.0) to version 10.2.0.3.0 by applying the 5337014 patchset(p5337014_10203_LINUX.zip). The patchset was applied to Oracle Clusterware and the OracleDatabase software. I also applied the one-off patchset - "BUNDLE Patch for Base Bug6000740" (MLR7 ON TOP OF 10.2.0.3) to the Oracle Clusterware and Oracle Databasesoftware. The procedures for installing both patchsets are not included in any of the parentarticle(s).

To maintain the current naming convention, the new Oracle RAC node to be added to theexisting cluster will be named linux3 (running a new instance named orcl3) making it a three-node cluster.

The new Oracle RAC node should have the same operating system version and installedpatches as the current two-node cluster.

Each node in the existing Oracle RAC cluster has a copy of the Oracle Clusterware and OracleDatabase software installed on their local disks. The current two-node Oracle RAC environmentdoes not use shared Oracle homes for the Clusterware or Database software.

The software owner for the Oracle Clusterware and Oracle Database installs will be "oracle". It

Eclipse plugin XML Editor30 day Free trial of Native XML Databases support

ORACLE - DBA Tips Corner 7/5/2015

http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle10gRAC/CLUSTER_20.shtml 1 / 41

is important that the UID and GID of the oracle user account be identical to that of the existingRAC nodes. For the purpose of this example, the oracle user account will be defined asfollows:[oracle@linux1 ~]$ id oracleuid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

The existing Oracle RAC 10g environment makes use of a clustered file system (OCFS2) tostore the two files required to be shared by Oracle Clusterware; namely the Oracle ClusterRegistry (OCR) file and the Voting Disk. Instructions for installing and adding the new OracleRAC node to the "live" OCFS2 file system will be included.

Automatic Storage Management (ASM) is being used as the file system and volume managerfor all Oracle physical database files (data, online redo logs, control files, archived redo logs)and a Flash Recovery Area. In addition to ASM, we will also be configuring ASMLib on the newOracle RAC node.

To add instances to an existing RAC database, Oracle Corporation recommends using theOracle cloning procedures which is described in the Oracle Universal Installer and OPatchUser's Guide. This article, however, uses manual procedures to add nodes and instances to theexisting Oracle RAC cluster. The manual procedures method described in this article involveextending the RAC database by first extending the Oracle Clusterware home to the new OracleRAC node and then extending the Oracle Database home. In other words, you extend thesoftware onto the new node in the same order as you installed the clusterware and Oracledatabase software components on the existing two-node RAC.

During the creation of the existing two-node cluster, the installation of Oracle Clusterware andthe Oracle Database software were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. The Oracle Universal Installer (OUI) on thatparticular node would then use the ssh and scp commands to run remote commands on andcopy files (the Oracle software) to all other nodes within the RAC cluster. The oracle useraccount on the node running the OUI (runInstaller) had to be trusted by all other nodes in theRAC cluster. This meant that the oracle user account had to run the secure shell commands(ssh or scp) on the Linux server executing the OUI (linux1) against all other Linux servers in thecluster without being prompted for a password. The same security requirements hold true forthis article. User equivalence will be configured so that the Oracle Clusterware and OracleDatabase software will be securely copied from linux1 to the new Oracle RAC node (linux3)using ssh and scp without being prompted for a password.

All shared disk storage for the existing Oracle RAC is based on iSCSI using a Network StorageServer; namely Openfiler Release 2.2 (respin 2). Powered by rPath Linux, Openfiler is a freebrowser-based network storage management utility that delivers file-based Network AttachedStorage (NAS) and block-based Storage Area Networking (SAN) in a single framework.Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of itsiSCSI capabilities to implement an inexpensive SAN for the shared storage componentsrequired by Oracle RAC 10g. This solution offers a low-cost alternative to fibre channel fortesting and educational purposes, but given the low-end hardware being used, it should not beused in a production environment.

These articles provide a low cost alternative for those who want to become familiar with OracleRAC 10g using commercial off the shelf components and downloadable software. Bear in mindthat these articles are provided for educational purposes only so the setup is kept simple todemonstrate ideas and concepts. For example, the disk mirroring configured in this article willbe setup on one physical disk only, while in practice that should be performed on multiplephysical drives. In addition, each Linux node will only be configured with two network cards one for the public network (eth0) and one for the private cluster interconnect "and" networkstorage server for shared iSCSI access (eth1). For a production RAC implementation, theprivate interconnect should be at least gigabit (or more) and "only" be used by Oracle to transferCluster Manager and Cache Fusion related data. A third dedicated network interface (i.e. eth2)should be configured on another gigabit network for access to the network storage server(Openfiler).

The following is a conceptual look at what the environment will look like after adding the third Oracle RAC node (linux3) tothe cluster. Click on the graphic below to enlarge the image:

Figure 1: Adding linux3 to the current Oracle RAC 10g Release 2 Environment

While this article provides comprehensive instructions for successfully adding a node to an existing OracleRAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article,users should also consult the following Oracle documents to gain a full understanding of alternativeconfiguration options, installation, and administration with Oracle RAC 10g. Oracle's official documentation siteis docs.oracle.com.

Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10gRelease 2 (10.2)



2 Day + Real Application Clusters Guide - 10g Release 2 (10.2)

Hardware and Costs

The hardware used in this article to build the third node (linux3) consists of a Linux workstation and components whichcan be purchased at many local computer stores or over the Internet.

Oracle RAC Node 3 - (linux3)

Dell Dimension 3000 Series

- Intel(R) Pentium(R) 4 Processor at 2.80GHz - 2GB DDR SDRAM (at 333MHz) - 60GB 7200 RPM Internal Hard Drive - Integrated Intel 3D AGP Graphics - Integrated 10/100 Ethernet - (Broadcom BCM4401) - CDROM (48X Max Variable) - 3.5" Floppy - No Keyboard, Monitor, or Mouse - (Connected to KVM Switch) US$300

1 - Ethernet LAN Card

Each Linux server for Oracle RAC should contain two NIC adapters. The Dell Dimensionincludes an integrated 10/100 Ethernet adapter that will be used to connect to the publicnetwork. The second NIC adapter will be used for the private network (RAC interconnectand Openfiler networked storage). Select the appropriate NIC adapter that is compatiblewith the maximum data transmission speed of the network switch to be used for theprivate network. For the purpose of this article, I used a Gigabit Ethernet switch (and1Gb Ethernet cards) for the private network.

Used for RAC interconnect to linux1, linux2 and Openfiler networked storage.

Gigabit Ethernet

Intel 10/100/1000Mbps PCI Desktop Adapter - (PWLA8391GT)US$35

2 - Network Cables

Category 5e patch cable - (Connect linux3 to public network) Category 5e patch cable - (Connect linux3 to interconnect ethernet switch)

US$5US$5

Total US$345

We are about to start the installation process. As we start to go into the details of the installation, it should be noted thatmost of the tasks within this document will need to be performed on the new Oracle RAC node (linux3). I will indicate atthe beginning of each section whether or not the task(s) should be performed on the new Oracle RAC node, the currentOracle RAC node(s), or on the network storage server (openfiler1).

Install the Linux Operating System

Perform the following installation on the new Oracle RAC node!

After procuring the required hardware, it is time to start the configuration process. The first task we need to perform is toinstall the Linux operating system. As already mentioned, this article will use CentOS 4.5. Although I have used Red HatFedora in the past, I wanted to switch to a Linux environment that would guarantee all of the functionality contained withOracle. This is where CentOS comes in. The CentOS project takes the Red Hat Enterprise Linux 4 source RPMs andcompiles them into a free clone of the Red Hat Enterprise Server 4 product. This provides a free and stable version of theRed Hat Enterprise Linux 4 (AS/ES) operating environment that I can now use for testing different Oracle configurations. Ihave moved away from Fedora as I need a stable environment that is not only free, but as close to the actual Oraclesupported operating system as possible. While CentOS is not the only project performing the same functionality, I tend tostick with it as it is stable and reacts fast with regards to updates by Red Hat.

Downloading CentOS

Use the links (below) to download CentOS 4.5. After downloading CentOS, you will then want to burn each ofthe ISO images to CD.

CentOS.org

CentOS-4.5-i386-bin1of4.iso (622 MB)CentOS-4.5-i386-bin2of4.iso (636 MB)CentOS-4.5-i386-bin3of4.iso (638 MB)CentOS-4.5-i386-bin4of4.iso (313 MB)

If you are downloading the above ISO files to a MS Windows machine, there aremany options for burning these images (ISO files) to a CD. You may already befamiliar with and have the proper software to burn images to CD. If you are notfamiliar with this process and do not have the required software to burn images to



CD, here are just two (of many) software packages that can be used:

UltraISO Magic ISO Maker

Installing CentOS

This section provides a summary of the screens used to install CentOS. For more detailed installationinstructions, it is possible to use the manuals from Red Hat Linux http://www.redhat.com/docs/manuals/. I wouldsuggest, however, that the instructions I have provided below be used for this Oracle RAC 10g configuration.

Before installing the Linux operating system on the new Oracle RAC node, youshould have the two NIC interfaces (cards) installed.

After downloading and burning the CentOS images (ISO files) to CD, insert CentOS Disk #1 into the newOracle RAC server (linux3 in this example), power it on, and answer the installation screen prompts as notedbelow.

Boot Screen

The first screen is the CentOS boot screen. At the boot: prompt, hit [Enter] to start the installationprocess.

Media Test

When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, themedia burning software would have warned us. After several seconds, the installer should thendetect the video card, monitor, and mouse. The installer then goes into GUI mode.

Welcome to CentOS

At the welcome screen, click [Next] to continue.

Language / Keyboard Selection

The next two screens prompt you for the Language and Keyboard settings. In almost all cases,you can accept the defaults. Make the appropriate selection for your configuration and click[Next] to continue.

Installation Type

Choose the [Custom] option and click [Next] to continue.

Disk Partitioning Setup

Select [Automatically partition] and click [Next] continue.

If there were a previous installation of Linux on this machine, the next screen will ask if you wantto "remove" or "keep" old partitions. Select the option to [Remove all partitions on this system].Also, ensure that the [hda] drive is selected for this installation. I also keep the checkbox [Review(and modify if needed) the partitions created] selected. Click [Next] to continue.

You will then be prompted with a dialog window asking if you really want to remove all partitions.Click [Yes] to acknowledge this warning.

Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automaticallyselected. For most automatic layouts, the installer will choose 100MB for /boot, double theamount of RAM (systems with < 2GB RAM) or an amount equal to RAM (systems with > 2GBRAM) for swap, and the rest going to the root (/) partition. Starting with EL 4, the installer willcreate the same disk configuration as just noted but will create them using the Logical VolumeManager (LVM). For example, it will partition the first hard drive (/dev/hda for my configuration)into two partitions one for the /boot partition (/dev/hda1) and the remainder of the diskdedicate to a LVM named VolGroup00 (/dev/hda2). The LVM Volume Group (VolGroup00) isthen partitioned into two LVM partitions - one for the root filesystem (/) and another for swap.

The main concern during the partitioning phase is to ensure enough swap space is allocated asrequired by Oracle (which is a multiple of the available RAM). The following is Oracle'srequirement for swap space:

Available RAM Swap Space RequiredBetween 1 GB and 2 GB 1.5 times the size of RAMBetween 2 GB and 8 GB Equal to the size of RAMMore than 8 GB .75 times the size of RAM

For the purpose of this install, I will accept all automatically preferred sizes. (Including 2GB forswap since I have 2GB of RAM installed.)



If for any reason, the automatic layout does not configure an adequate amount of swap space,you can easily change that from this screen. To increase the size of the swap partition, [Edit] thevolume group VolGroup00. This will bring up the "Edit LVM Volume Group: VolGroup00" dialog.First, [Edit] and decrease the size of the root file system (/) by the amount you want to add to theswap partition. For example, to add another 512MB to swap, you would decrease the size of theroot file system by 512MB (i.e. 36,032MB - 512MB = 35,520MB). Now add the space youdecreased from the root file system (512MB) to the swap partition. When completed, click [OK]on the "Edit LVM Volume Group: VolGroup00" dialog.

Once you are satisfied with the disk layout, click [Next] to continue.

Boot Loader Configuration

The installer will use the GRUB boot loader by default. To use the GRUB boot loader, accept alldefault values and click [Next] to continue.

Network Configuration

I made sure to install both NIC interfaces (cards) in the new Linux machine before starting theoperating system installation. This screen should have successfully detected each of thenetwork devices.

First, make sure that each of the network devices are checked to [Active on boot]. The installermay choose to not activate eth1 by default.

Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP addresses forboth eth0 and eth1 and that is OK. Configure eth1 (the interconnect and storage network) on adifferent subnet than eth0 (the public network):

eth0:- Check OFF the option to [Configure using DHCP]- Leave the [Activate on boot] checked ON- IP Address: 192.168.1.107- Netmask: 255.255.255.0

eth1:- Check OFF the option to [Configure using DHCP]- Leave the [Activate on boot] checked ON- IP Address: 192.168.2.107- Netmask: 255.255.255.0

Continue by setting your hostname manually. I used "linux3" for this new Oracle RAC node.Finish this dialog off by supplying your gateway and DNS servers.

Firewall

On this screen, make sure to select [No firewall]. Also under the option to "Enable SELinux?",select [Disabled] and click [Next] to continue.

You will be prompted with a warning dialog about not setting the firewall. If this occurs, simply hit[Proceed] to continue.

Additional Language Support / Time Zone

The next two screens allow you to select additional language support and time zone information.In almost all cases, you can accept the defaults. Make the appropriate selection for yourconfiguration and click [Next] to continue.

Set Root Password

Select a root password and click [Next] to continue.

Package Group Selection

Scroll down to the bottom of this screen and select [Everything] under the "Miscellaneous"section. Click [Next] to continue.

Please note that the installation of Oracle does not require all Linux packages to be installed.My decision to install all packages was for the sake of brevity. Please see section "Pre-Installation Tasks for Oracle10g Release 2" for a more detailed look at the critical packagesrequired for a successful Oracle installation.

Also note that with some RHEL 4 distributions, you will not get the "Package Group Selection"screen by default. There, you are asked to simply "Install default software packages" or"Customize software packages to be installed". Select the option to "Customize softwarepackages to be installed" and click [Next] to continue. This will then bring up the "PackageGroup Selection" screen. Now, scroll down to the bottom of this screen and select [Everything]under the "Miscellaneous" section. Click [Next] to continue.

About to Install

This screen is basically a confirmation screen. Click [Next] on this screen and then the



[Continue] button on the dialog box to start the installation. During the installation process, youwill be asked to switch disks to Disk #2, Disk #3, and then Disk #4.

Note that with CentOS 4.5, the installer would ask to switch to Disk #2, Disk #3, Disk #4, Disk #1,and then back to Disk #4.

Graphical Interface (X) Configuration

With most RHEL 4 distributions (not the case with CentOS 4.5), when the installation iscomplete, the installer will attempt to detect your video hardware. Ensure that the installer hasdetected and selected the correct video hardware (graphics card and monitor) to properly usethe X Windows server. You will continue with the X configuration in the next serveral screens.

Congratulations

And that's it. You have successfully installed CentOS on the new Oracle RAC node (linux3). Theinstaller will eject the CD from the CD-ROM drive. Take out the CD and click [Reboot] to rebootthe system.

When the system boots into Linux for the first time, it will prompt you with another Welcomescreen. The following wizard allows you to configure the date and time, add any additional users,test the sound card, and to install any additional CDs. The only screen I care about is the timeand date (and if you are using CentOS 4.x, the monitor/display settings). As for the others,simply run through them as there is nothing additional that needs to be installed (at this pointanyways!). If everything was successful, you should now be presented with the login screen.

Network Configuration

Perform the following network configuration tasks on the new Oracle RACnode!

Introduction to Network Settings

Although we configured several of the network settings during the installation of CentOS, it is important to notskip this section as it contains critical steps that are required for a successful RAC environment.

During the Linux O/S install we already configured the IP address and host name for the new Oracle RACnode. We now need to configure the /etc/hosts file as well as adjusting several of the network settings for theinterconnect.

All nodes in the RAC cluster should have one static IP address for the public network and one static IP addressfor the private cluster interconnect. The private interconnect should only be used by Oracle to transfer ClusterManager and Cache Fusion related data along with data for the network storage server (Openfiler). Note thatOracle does not support using the public network interface for the interconnect. You must have one networkinterface for the public network and another network interface for the private interconnect. For a productionRAC implementation, the interconnect should be at least gigabit (or more) and only be used by Oracle as wellas having the network storage server (Openfiler) on a separate gigabit network.

Configuring Public and Private Network

With the new Oracle RAC node, we need to configure the network for access to the public network as well asthe private interconnect.

The easiest way to configure network settings in Red Hat Linux is with the program Network Configuration.This application can be started from the command-line as the "root" user account as follows:

# su -# /usr/bin/system-config-network &

Do not use DHCP naming for the public IP address or the interconnects - we needstatic IP addresses!

Using the Network Configuration application, we will need to configure both NIC devices as well as the/etc/hosts file on all nodes in the RAC cluster. Both of these tasks can be completed using the NetworkConfiguration GUI.

Please note that for the purpose of this example configuration the /etc/hosts entries will be the same for allthree Oracle RAC nodes (linux1, linux2, and linux3) as well as the network storage server (openfiler1):

Our example configuration will use the following settings for all nodes:

Oracle RAC Node 3 - (linux3)Device IP Address Subnet Gateway Purposeeth0 192.168.1.107 255.255.255.0 192.168.1.1 Connects linux3 to the public network

eth1 192.168.2.107 255.255.255.0 Connects linux3 (interconnect) tolinux1/linux2 (linux1-priv/linux2-priv)



linux1/linux2 (linux1-priv/linux2-priv)/etc/hosts127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0)192.168.1.100 linux1192.168.1.101 linux2192.168.1.107 linux3

# Private Interconnect - (eth1)192.168.2.100 linux1-priv192.168.2.101 linux2-priv192.168.2.107 linux3-priv

# Public Virtual IP (VIP) addresses - (eth0:1)192.168.1.200 linux1-vip192.168.1.201 linux2-vip192.168.1.207 linux3-vip

# Private Storage Network for Openfiler192.168.1.195 openfiler1192.168.2.195 openfiler1-priv


eth1 192.168.2.101 255.255.255.0 Connects linux2 (interconnect) tolinux1/linux3 (linux1-priv/linux3-priv)/etc/hosts127.0.0.1 localhost.localdomain localhost






eth1 192.168.2.100 255.255.255.0 Connects linux1 (interconnect) tolinux2/linux3 (linux2-priv/linux3-priv)/etc/hosts127.0.0.1 localhost.localdomain localhost





In the screen shots below, only the new Oracle RAC node (linux3) is shown. Ensure that the /etc/hosts file isupdated on all participating nodes to access the new Oracle RAC node!

Figure 2: Network Configuration Screen - Node 3 (linux3)

Figure 3: Ethernet Device Screen - eth0 (linux3)



Figure 4: Ethernet Device Screen - eth1 (linux3)

Figure 5: Network Configuration Screen - /etc/hosts (linux3)

Once the network is configured, you can use the ifconfig command to verify everything is working. Thefollowing example is from the new Oracle RAC node linux3:

# /sbin/ifconfig -aeth0 Link encap:Ethernet HWaddr 00:1E:2A:37:6B:9E inet addr:192.168.1.107 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21e:2aff:fe37:6b9e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1167677 errors:0 dropped:0 overruns:0 frame:0 TX packets:1842517 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:576629131 (549.9 MiB) TX bytes:2143836310 (1.9 GiB) Interrupt:209 Base address:0xef00

eth1 Link encap:Ethernet HWaddr 00:0E:0C:C0:78:64 inet addr:192.168.2.107 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20e:cff:fec0:7864/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:48 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4782 (4.6 KiB) TX bytes:5564 (5.4 KiB) Base address:0xdd80 Memory:fe9c0000-fe9e0000

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2034 errors:0 dropped:0 overruns:0 frame:0 TX packets:2034 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2386686 (2.2 MiB) TX bytes:2386686 (2.2 MiB)

sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Verify Network Access to All Nodes

Verify that the new Oracle RAC node has access to the public and private network for all current nodes. Fromlinux3:# ping -c 1 linux1 | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

# ping -c 1 linux1-priv | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

# ping -c 1 linux2 | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

# ping -c 1 linux2-priv | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

# ping -c 1 openfiler1 | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

# ping -c 1 openfiler1-priv | grep '1 packets transmitted'1 packets transmitted, 1 received, 0% packet loss, time 0ms

Confirm the RAC Node Name is Not Listed in Loopback Address

Ensure that the new Oracle RAC node (linux3) is not included for the loopback address in the /etc/hostsfile. If the machine name is listed in the in the loopback address entry as below: 127.0.0.1 linux3 localhost.localdomain localhost

it will need to be removed as shown below: 127.0.0.1 localhost.localdomain localhost

If the RAC node name is listed for the loopback address, you will receive thefollowing error during the RAC installation:ORA-00603: ORACLE server session terminated by fatal error

orORA-29702: error occurred in Cluster Group Service operation



Confirm localhost is defined in the /etc/hosts file for the loopback address

Ensure that the entry for localhost.localdomain and localhost are included for the loopback address in the/etc/hosts file for the new Oracle RAC node: 127.0.0.1 localhost.localdomain localhost

If an entry does not exist for localhost in the /etc/hosts file, Oracle Clusterwarewill be unable to start the application resources notably the ONS process. Theerror would indicate "Failed to get IP for localhost" and will be written to the logfile for ONS. For example:CRS-0215 could not start resource 'ora.linux3.ons'. Check log file"/u01/app/crs/log/linux3/racg/ora.linux3.ons.log"for more details.

The ONS log file will contain lines similar to the following:

Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996,2005 Oracle. All rights reserved.2007-04-14 13:10:02.729: [RACG][3086871296][13316][3086871296][ora.linux3.ons]: Failed to get IPfor localhost (1)Failed to get IP for localhost (1)Failed to get IP for localhost (1)onsctl: ons failed to start...

Adjusting Network Settings

With Oracle 9.2.0.1 and onwards, Oracle now makes use of UDP as the default protocol on Linux for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer transfers betweeninstances within the RAC cluster.

Oracle strongly suggests to adjust the default and maximum send buffer size (SO_SNDBUF socket option) to 256KB, and the default and maximum receive buffer size (SO_RCVBUF socket option) to 256 KB.

The receive buffers are used by TCP and UDP to hold received data until it is read by the application. Thereceive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window.This means that datagrams will be discarded if they don't fit in the socket receive buffer. This could cause thesender to overwhelm the receiver.

The default and maximum window size can be changed without a reboot. Add the following entries to the/etc/sysctl.conf file on the new Oracle RAC node:

# +---------------------------------------------------------+# | ADJUSTING NETWORK SETTINGS |# +---------------------------------------------------------+# | With Oracle 9.2.0.1 and onwards, Oracle now makes use |# | of UDP as the default protocol on Linux for |# | inter-process communication (IPC), such as Cache Fusion |# | and Cluster Manager buffer transfers between instances |# | within the RAC cluster. Oracle strongly suggests to |# | adjust the default and maximum receive buffer size |# | (SO_RCVBUF socket option) to 256 KB, and the default |# | and maximum send buffer size (SO_SNDBUF socket option) |# | to 256 KB. The receive buffers are used by TCP and UDP |# | to hold received data until it is read by the |# | application. The receive buffer cannot overflow because |# | the peer is not allowed to send data beyond the buffer |# | size window. This means that datagrams will be |# | discarded if they don't fit in the socket receive |# | buffer. This could cause the sender to overwhelm the |# | receiver. |# +---------------------------------------------------------+

# +---------------------------------------------------------+# | Default setting in bytes of the socket "receive" buffer |# | which may be set by using the SO_RCVBUF socket option. |# +---------------------------------------------------------+net.core.rmem_default=262144

# +---------------------------------------------------------+# | Maximum setting in bytes of the socket "receive" buffer |# | which may be set by using the SO_RCVBUF socket option. |# +---------------------------------------------------------+net.core.rmem_max=262144

# +---------------------------------------------------------+# | Default setting in bytes of the socket "send" buffer |# | which may be set by using the SO_SNDBUF socket option. |# +---------------------------------------------------------+net.core.wmem_default=262144

# +---------------------------------------------------------+# | Maximum setting in bytes of the socket "send" buffer |# | which may be set by using the SO_SNDBUF socket option. |# +---------------------------------------------------------+net.core.wmem_max=262144



Then, ensure that each of these parameters are truly in effect by running the following command on the newOracle RAC node:# sysctl -pnet.ipv4.ip_forward = 0net.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.accept_source_route = 0kernel.sysrq = 0kernel.core_uses_pid = 1net.core.rmem_default = 262144net.core.rmem_max = 262144net.core.wmem_default = 262144net.core.wmem_max = 262144

Check and turn off UDP ICMP rejections:

During the Linux installation process, I indicated to not configure the firewall option. (By default the option toconfigure a firewall is selected by the installer.) This has burned me several times so I like to do a double-check that the firewall option is not configured and to ensure udp ICMP filtering is turned off.

If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will crash after severalminutes of running. When the Oracle Clusterware process fails, you will have something similar to the followingin the _evmocr.log file:

08/29/2005 22:17:19oac_init:2: Could not connect to server, clsc retcode = 908/29/2005 22:17:19a_init:12!: Client init unsuccessful : [32]ibctx:1:ERROR: INVALID FORMATproprinit:problem reading the bootblock or superbloc 22

When experiencing this type of error, the solution is to remove the udp ICMP (iptables) rejection rule - or tosimply have the firewall option turned off. The Oracle Clusterware software will then start to operate normallyand not crash. The following commands should be executed as the root user account:

1. Check to ensure that the firewall option is turned off. If the firewall option is stopped (like it is in myexample below) you do not have to proceed with the following steps.# /etc/rc.d/init.d/iptables statusFirewall is stopped.

2. If the firewall option is operating you will need to first manually disable UDP ICMP rejections:# /etc/rc.d/init.d/iptables stop

Flushing firewall rules: [ OK ]Setting chains to policy ACCEPT: filter [ OK ]Unloading iptables modules: [ OK ]

3. Then, to turn UDP ICMP rejections off for next server reboot (which should always be turned off):# chkconfig iptables off

Configure Network Security on the Openfiler Storage Server

Perform the following configuration tasks on the network storage server(openfiler1)!

With the network now setup, the next step is to configure network access in Openfiler so that the new Oracle RAC node(linux3) has permissions to the shared iSCSI volumes used in the current Oracle RAC 10g environment. For the purposeof this example, all iSCSI traffic will use the private network interface eth1 which in this article is on the 192.168.2.0network.

Openfiler administration is performed using the Openfiler Storage Control Center a browser based tool over an httpsconnection on port 446. For example:

https://openfiler1:446/

From the Openfiler Storage Control Center home page, login as an administrator. The default administration logincredentials for Openfiler are:

Username: openfilerPassword: password

The first page the administrator sees is the [Accounts] / [Authentication] screen. Configuring user accounts and groups isnot necessary for this article and will therefore not be discussed.

Services

This article assumes that the current Oracle RAC 10g environment is operational and therefore the iSCSIservices should already be enabled within Openfiler.



To verify the iSCSI services are running, use the Openfiler Storage Control Center and navigate to [Services] /[Enable/Disable]:

Figure 6: Verify iSCSI Services are Enabled

Another method is to SSH into the Openfiler server and verify the iscsi-target service is running:

[root@openfiler1 ~]# service iscsi-target statusietd (pid 3784) is running...

Network Access Restriction

The next step is to configure network access in Openfiler so that the new Oracle RAC node (linux3) haspermissions to the shared iSCSI volumes used in the current Oracle RAC 10g environment.

Again, this task can be completed using the Openfiler Storage Control Center by navigating to [General] /[Local Networks]. The Local Networks screen allows an administrator to setup networks and/or hosts that willbe allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we willwant to add the new Oracle RAC node individually rather than allowing the entire 192.168.2.0 network haveaccess to Openfiler resources.

When entering the new Oracle RAC node, note that the 'Name' field is just a logical name used for referenceonly. As a convention when entering nodes, I simply use the node name defined for that IP address. Next, whenentering the actual node in the 'Network/Host' field, always use it's IP address even though its host name mayalready be defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class C network,use a subnet mask of 255.255.255.255.

It is important to remember that you will be entering the IP address of the private network (eth1) for the newOracle RAC node.

The following image shows the results of adding the new Oracle RAC node linux3 to the local networkconfiguration:

Figure 7: Configure Openfiler Host Access for new Oracle RAC Node

Current Logical iSCSI Volumes

The current Openfiler configuration contains five logical iSCSI volumes in a single volume group named rac1.

iSCSI / Logical Volumes in Volume Group rac1Volume Name Volume Description Required Space (MB) Filesystem Typecrs Oracle Clusterware 2,048 iSCSIasm1 Oracle ASM Volume 1 118,720 iSCSIasm2 Oracle ASM Volume 2 118,720 iSCSIasm3 Oracle ASM Volume 3 118,720 iSCSIasm4 Oracle ASM Volume 4 118,720 iSCSI

To view the available iSCSI volumes from within the Openfiler Storage Control Center, navigate to [Volumes] /[List of Existing Volumes]. There we will see all five logical volumes within the volume group rac1:

Figure 8: Current Logical (iSCSI) Volumes

Grant Access Rights to New Logical Volumes

Before an iSCSI client can have access to any of the iSCSI volumes, it needs to be granted the appropriatepermissions. In this section, we need to grant access to each of the five local iSCSI volumes to the new OracleRAC node linux3.

From the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing Volumes]. This will presentthe screen shown in the previous section. For each of the five logical volumes, click on the 'Edit' link (under theProperties column). This will bring up the 'Edit properties' screen for that volume. Scroll to the bottom of thisscreen; change the access for host linux3-priv from 'Deny' to 'Allow' and click the 'Update' button. Performthis task for all five logical volumes.

Figure 9: Grant Host Access to Logical (iSCSI) Volumes

Configure the iSCSI Initiator

Configure the iSCSI initiator on the new Oracle RAC node!



An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) isavailable. In our case, the clients are the three Oracle RAC nodes, (linux1, linux2, and linux3), running Red Hat 4.

In this section we will be configuring the iSCSI initiator on the new Oracle RAC node linux3. This involves configuring the/etc/iscsi.conf file on the new Oracle RAC node with the name of the network storage server (openfiler1) so it candiscover the current iSCSI volumes.

iSCSI (initiator) service

On the new Oracle RAC node, we have to make sure the iSCSI (initiator) service is up and running. If notinstalled as part of the operating system setup, the iscsi-initiator-utils RPM (i.e. iscsi-initiator-utils-4.0.3.0-5.i386.rpm) should be downloaded and installed on the new Oracle RAC node.

The new Oracle RAC node must have the iscsi-initiator-utils RPM installed. To determine ifthis package is installed, perform the following:# rpm -qa | grep iscsiiscsi-initiator-utils-4.0.3.0-5

If not installed, the iscsi-initiator-utils RPM package can be found on disk 3 of 4 of theRHEL4 Update 5 distribution or downloaded from one of the Internet RPM resources.

Use the following command to install the iscsi-initiator-utils RPM package if not present:

# rpm -Uvh iscsi-initiator-utils-4.0.3.0-5.i386.rpmwarning: iscsi-initiator-utils-4.0.3.0-5.i386.rpm: V3 DSA signature: NOKEY, key ID 443e1821Preparing... ########################################### [100%] 1:iscsi-initiator-utils ########################################### [100%]

After verifying that the iscsi-initiator-utils RPM is installed, the only configuration step required on the newOracle RAC node (iSCSI client) is to specify the network storage server (iSCSI server) in the /etc/iscsi.conffile. Edit the /etc/iscsi.conf file and include an entry for DiscoveryAddress which specifies the hostname ofthe Openfiler network storage server. In our case that was:

...DiscoveryAddress=openfiler1-priv...

After making the change to the /etc/iscsi.conf file on the new Oracle RAC node, we can start (or restart) theiscsi initiator service on that node:# service iscsi restartSearching for iscsi-based multipath mapsFound 0 mapsStopping iscsid: iscsid not running

Checking iscsi config: [ OK ]Loading iscsi driver: [ OK ]Starting iscsid: [ OK ]

We should also configure the iSCSI service to be active across machine reboots for the new Oracle RACnode. The Linux command chkconfig can be used to achieve that as follows:# chkconfig --level 345 iscsi on

Discovering iSCSI Targets

The iSCSI initiator service should now be configured and started on the new Oracle RAC node. In the parent tothis article ("Building an Inexpensive Oracle RAC 10g Release 2 on Linux - (CentOS 4.5 / iSCSI)"), we neededgo through the arduous task of mapping the iSCSI target names discovered from Openfiler to the local SCSIdevice name on one of the Oracle RAC nodes. Given that all five logical iSCSI volumes were partitioned andformatted with labels in that article, we don't have to perform that task again. Note that one of the iSCSIvolumes was formatted and labeled using OCFS2 while the other four were labeled for use by ASM.

In this section, we simply want to verify that the new Oracle RAC node was able to successfully discover thefive logical iSCSI volumes on the Openfiler server.

When the Openfiler server publishes available iSCSI targets, configured clients get the message that newiSCSI disks are now available. This happens when the iscsi-target service gets started/restarted on theOpenfiler server or when the iSCSI initiator service is started/restarted on the client. We would see somethinglike this in the client's /var/log/messages file:

...Jan 21 16:41:29 linux3 iscsi: iscsid startup succeededJan 21 16:41:29 linux3 iscsid[13822]: Connected to Discovery Address 192.168.2.195Jan 21 16:41:29 linux3 kernel: iscsi-sfnet:host0: Session establishedJan 21 16:41:29 linux3 kernel: iscsi-sfnet:host2: Session establishedJan 21 16:41:29 linux3 kernel: iscsi-sfnet:host1: Session establishedJan 21 16:41:29 linux3 kernel: scsi0 : SFNet iSCSI driverJan 21 16:41:29 linux3 kernel: scsi2 : SFNet iSCSI driverJan 21 16:41:29 linux3 kernel: scsi1 : SFNet iSCSI driverJan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04Jan 21 16:41:29 linux3 kernel: SCSI device sda: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:29 linux3 kernel: SCSI device sda: drive cache: write through



Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04Jan 21 16:41:29 linux3 kernel: SCSI device sda: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:29 linux3 kernel: iscsi-sfnet:host3: Session establishedJan 21 16:41:29 linux3 kernel: iscsi-sfnet:host4: Session establishedJan 21 16:41:29 linux3 kernel: scsi3 : SFNet iSCSI driverJan 21 16:41:29 linux3 kernel: SCSI device sda: drive cache: write throughJan 21 16:41:29 linux3 kernel: sda: unknown partition tableJan 21 16:41:29 linux3 kernel: Attached scsi disk sda at scsi0, channel 0, id 0, lun 0Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0Jan 21 16:41:29 linux3 scsi.agent[13934]: disk at /devices/platform/host0/target0:0:0/0:0:0:0Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04Jan 21 16:41:29 linux3 kernel: scsi4 : SFNet iSCSI driverJan 21 16:41:29 linux3 kernel: SCSI device sdb: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:29 linux3 kernel: Vendor: Openfile Model: Virtual disk Rev: 0Jan 21 16:41:29 linux3 kernel: Type: Direct-Access ANSI SCSI revision: 04Jan 21 16:41:29 linux3 kernel: SCSI device sdb: drive cache: write throughJan 21 16:41:29 linux3 scsi.agent[13983]: disk at /devices/platform/host2/target2:0:0/2:0:0:0Jan 21 16:41:29 linux3 scsi.agent[13996]: disk at /devices/platform/host3/target3:0:0/3:0:0:0Jan 21 16:41:30 linux3 kernel: SCSI device sdb: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sdb: drive cache: write throughJan 21 16:41:30 linux3 kernel: sdb: unknown partition tableJan 21 16:41:30 linux3 kernel: Attached scsi disk sdb at scsi2, channel 0, id 0, lun 0Jan 21 16:41:30 linux3 kernel: SCSI device sdc: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sdc: drive cache: write throughJan 21 16:41:30 linux3 kernel: SCSI device sdc: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sdc: drive cache: write throughJan 21 16:41:30 linux3 kernel: sdc: unknown partition tableJan 21 16:41:30 linux3 kernel: Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0Jan 21 16:41:30 linux3 kernel: SCSI device sdd: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sdd: drive cache: write throughJan 21 16:41:30 linux3 kernel: SCSI device sdd: 243138560 512-byte hdwr sectors (124487 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sdd: drive cache: write throughJan 21 16:41:30 linux3 kernel: sdd: unknown partition tableJan 21 16:41:30 linux3 kernel: Attached scsi disk sdd at scsi1, channel 0, id 0, lun 0Jan 21 16:41:30 linux3 kernel: SCSI device sde: 4194304 512-byte hdwr sectors (2147 MB)Jan 21 16:41:30 linux3 scsi.agent[14032]: disk at /devices/platform/host4/target4:0:0/4:0:0:0Jan 21 16:41:30 linux3 scsi.agent[14045]: disk at /devices/platform/host1/target1:0:0/1:0:0:0Jan 21 16:41:30 linux3 kernel: SCSI device sde: drive cache: write throughJan 21 16:41:30 linux3 kernel: SCSI device sde: 4194304 512-byte hdwr sectors (2147 MB)Jan 21 16:41:30 linux3 kernel: SCSI device sde: drive cache: write throughJan 21 16:41:30 linux3 kernel: sde: unknown partition tableJan 21 16:41:30 linux3 kernel: Attached scsi disk sde at scsi4, channel 0, id 0, lun 0...

The above entries show that the client (linux3) was able to establish the iSCSI sessions with the iSCSIstorage server (openfiler1-priv at 192.168.2.195).

Another method not only checks for the existence of the iSCSI volumes, but also displays how the local SCSIdevice names map to iSCSI targets' host IDs and LUNs. Use the following script which was provided by MartinJones to display these mappings:

iscsi-ls-map.sh

# ---------------------# FILE: iscsi-ls-map.sh# ---------------------

RUN_USERID=rootexport RUN_USERID

RUID=ìd | awk -F$ '{print $2}'|awk -F$ '{print $1}'ìf [[ ${RUID} != "$RUN_USERID" ]];then echo " " echo "You must be logged in as $RUN_USERID to run this script." echo "Exiting script." echo " " exit 1fi

dmesg | grep "Âttach" \ | awk -F" " '{ print "/dev/"$4 " " $6 }' \ | sed -e 's/,//' | sed -e 's/scsi//' \ | sort -n -k2 \ | sed -e '/disk1/d' > /tmp/tmp_scsi_dev

iscsi-ls | egrep -e "TARGET NAME" -e "HOST ID" \ | awk -F" " '{ if ($0 ~ /^TARGET.*/) printf $4; if ( $0 ~ /^HOST/) printf " %s\n",$4}' \ | sort -n -k2 \ | cut -d':' -f2- \ | cut -d'.' -f2- > /tmp/tmp_scsi_targets

join -t" " -1 2 -2 2 /tmp/tmp_scsi_dev /tmp/tmp_scsi_targets > MAP

echo "Host / SCSI ID SCSI Device Name iSCSI Target Name"echo "---------------- ----------------------- -----------------"

cat MAP | sed -e 's/ / /g'

rm -f MAP

Example run:



# ./iscsi-ls-map.shHost / SCSI ID SCSI Device Name iSCSI Target Name---------------- ------------------------ -----------------0 /dev/sda asm41 /dev/sdd asm32 /dev/sdb asm23 /dev/sdc asm14 /dev/sde crs

Create "oracle" User and Directories

Perform the following tasks on the new Oracle RAC node!

I will be using the Oracle Cluster File System, Release 2 (OCFS2) to store the files required to be shared for the OracleClusterware software. When using OCFS2, the UID of the UNIX user "oracle" and GID of the UNIX group "oinstall" mustbe the same on all of the Oracle RAC nodes in the cluster. If either the UID or GID are different, the files on the OCFS2 filesystem will show up as "unowned" or may even be owned by a different user. For this article and its parent article, I will use501 for the "oracle" UID and 501 for the "oinstall" GID.

Note that members of the UNIX group oinstall are considered the "owners" of the Oracle software. Members of the dbagroup can administer Oracle databases, for example starting up and shutting down databases. In this article, we arecreating the oracle user account to have both responsibilities!

This guide adheres to the Optimal Flexible Architecture (OFA) for namingconventions used in creating the directory structure.

Create Group and User for Oracle

Lets start this section by creating the UNIX oinstall and dba group and oracle user account:

# groupadd -g 501 oinstall# groupadd -g 502 dba# useradd -m -u 501 -g oinstall -G dba -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle# id oracleuid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)

Set the password for the oracle account:# passwd oracleChanging password for user oracle.New UNIX password: xxxxxxxxxxxRetype new UNIX password: xxxxxxxxxxxpasswd: all authentication tokens updated successfully.

Verify That the User nobody Exists

Before installing the Oracle software, complete the following procedure to verify that the user nobody exists onthe system:

1. To determine if the user exists, enter the following command:# id nobodyuid=99(nobody) gid=99(nobody) groups=99(nobody)

If this command displays information about the nobody user, then you do not have to create that user.

2. If the user nobody does not exist, then enter the following command to create it:# /usr/sbin/useradd nobody

Create the Oracle Base Directory

The next step is to create a new directory that will be used to store the Oracle Database software. Whenconfiguring the oracle user's environment (later in this section) we will be assigning the location of thisdirectory to the $ORACLE_BASE environment variable.

The following assumes that the directories are being created in the root file system. Please note that this isbeing done for the sake of simplicity and is not recommended as a general practice. Normally, thesedirectories would be created on a separate file system.

After the directory is created, you must then specify the correct owner, group, and permissions for it. Performthe following on the new Oracle RAC node:

# mkdir -p /u01/app/oracle# chown -R oracle:oinstall /u01/app/oracle# chmod -R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

/u01 owned by root.



/u01/app owned by root./u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and permissionsenables the OUI to create the oraInventory directory, in the path /u01/app/oracle/oraInventory.

Create the Oracle Clusterware Home Directory

Next, create a new directory that will be used to store the Oracle Clusterware software. When configuring theoracle user's environment (later in this section) we will be assigning the location of this directory to the$ORA_CRS_HOME environment variable.

As noted in the previous section, the following assumes that the directories are being created in the root filesystem. This is being done for the sake of simplicity and is not recommended as a general practice. Normally,these directories would be created on a separate file system.

After the directory is created, you must then specify the correct owner, group, and permissions for it. Performthe following on the new Oracle RAC node:

# mkdir -p /u01/app/crs# chown -R oracle:oinstall /u01/app/crs# chmod -R 775 /u01/app/crs

At the end of this procedure, you will have the following:

/u01 owned by root./u01/app owned by root./u01/app/crs owned by oracle:oinstall with 775 permissions. These permissions are required forOracle Clusterware installation and are changed during the installation process.

Create Mount Point for OCFS2 / Clusterware

Let's now create the mount point for the Oracle Cluster File System, Release 2 (OCFS2) that will be used tostore the two Oracle Clusterware shared files.

Perform the following on the new Oracle RAC node:

# mkdir -p /u02# chown -R oracle:oinstall /u02# chmod -R 775 /u02

Create Login Script for oracle User Account

To ensure that the environment is setup correctly for the "oracle" UNIX userid on the new Oracle RAC node,use the following .bash_profile:

When you are setting the Oracle environment variables for each Oracle RACnode, ensure to assign each RAC node a unique Oracle SID!

For this example, I used:

linux1 : ORACLE_SID=orcl1linux2 : ORACLE_SID=orcl2linux3 : ORACLE_SID=orcl3

Login to the new Oracle RAC node as the oracle user account:

# su - oracle

.bash_profile for Oracle User# .bash_profile

# Get the aliases and functionsif [ -f ~/.bashrc ]; then . ~/.bashrcfi

alias ls="ls -FA"alias s="screen -DRRS iPad -t iPad"

export JAVA_HOME=/usr/local/java

# User specific environment and startup programsexport ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1export ORA_CRS_HOME=/u01/app/crsexport ORACLE_PATH=$ORACLE_BASE/dba_scripts/sql:.:$ORACLE_HOME/rdbms/adminexport CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2, orcl3,...)export ORACLE_SID=orcl3

export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/binexport PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/binexport PATH=${PATH}:$ORACLE_BASE/dba_scripts/bin



export ORACLE_TERM=xtermexport TNS_ADMIN=$ORACLE_HOME/network/adminexport ORA_NLS10=$ORACLE_HOME/nls/dataexport NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"export LD_LIBRARY_PATH=$ORACLE_HOME/libexport LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/libexport LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/libexport CLASSPATH=$ORACLE_HOME/JREexport CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlibexport CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlibexport CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlibexport THREADS_FLAG=nativeexport TEMP=/tmpexport TMPDIR=/tmp

Configure the Linux Server for Oracle


The kernel parameters and shell limits discussed in this section will need to be defined on the new Oracle RAC nodeevery time the machine is booted. This section will not go into great depth in explaining the purpose of those kernelparameters that are required by Oracle (These parameters are described in detail in the parent to this article). Providedin this section, however, are instructions on how to set all required kernel parameters for Oracle and how to have themenabled when the node boots. Further instructions for configuring kernel parameters in a startup script (/etc/sysctl.conf)is included in the section "All Startup Commands for New Oracle RAC Node".

Swap Space Considerations

Installing Oracle10g Release 2 requires a minimum of 512MB of memory. (An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang" or"die")

To check the amount of memory you have, type:# cat /proc/meminfo | grep MemTotalMemTotal: 2074428 kB

To check the amount of swap you have allocated, type:# cat /proc/meminfo | grep SwapTotalSwapTotal: 2031608 kB

If you have less than 512MB of memory (between your RAM and SWAP), you can add temporary swap space bycreating a temporary swap file. This way you do not have to use a raw device or even more drastic, rebuild yoursystem.

As root, make a file that will act as additional swap space, let's say about 500MB: # dd if=/dev/zero of=tempswap bs=1k count=500000

Now we should change the file permissions: # chmod 600 tempswap

Finally we format the "partition" as swap and add it to the swap space: # mke2fs tempswap# mkswap tempswap# swapon tempswap

Configuring Kernel Parameters and Shell Limits

The kernel parameters and shell limits presented in this section are recommended values only as documentedby Oracle. For production database systems, Oracle recommends that you tune these values to optimize theperformance of the system.

On the new Oracle RAC node, verify that the kernel parameters described in this section are set to valuesgreater than or equal to the recommended values. Also note that when setting the four semaphore values thatall four values need to be entered on one line.

Setting Shared Memory / Semaphores / File Handles / Local IP Range

Set the following kernel parameters in the /etc/sysctl.conf file on the new Oracle RAC node.cat >> /etc/sysctl.conf

# | SHARED MEMORY |# +---------------------------------------------------------+kernel.shmmax=2147483648

# +---------------------------------------------------------+# | SEMAPHORES |# | ---------- |# | |# | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value |# | |# +---------------------------------------------------------+kernel.sem=250 32000 100 128

# +---------------------------------------------------------+# | FILE HANDLES |# ----------------------------------------------------------+fs.file-max=65536

# +---------------------------------------------------------+# | LOCAL IP RANGE |# ----------------------------------------------------------+net.ipv4.ip_local_port_range=1024 65000EOF

Setting Shell Limits for the oracle User

To improve the performance of the software on Linux systems, Oracle recommends you increase the followingshell limits for the oracle user:

Shell Limit Item in limits.conf Hard LimitMaximum number of open file descriptors nofile 65536Maximum number of processes available to a single user nproc 16384

To make these changes, run the following as root:

cat >> /etc/security/limits.conf /etc/pam.d/login /etc/profile /etc/csh.login

net.core.rmem_max = 262144net.core.wmem_default = 262144net.core.wmem_max = 262144kernel.shmmax = 2147483648kernel.sem = 250 32000 100 128fs.file-max = 65536net.ipv4.ip_local_port_range = 1024 65000

Setting the Correct Date and Time on the new Oracle RAC Node

When adding the new Oracle RAC node to the cluster, the Oracle Universal Installer (OUI) copies the OracleClusterware and Oracle Database software from the source RAC node (linux1 in this article) to the new nodein the cluster (linux3). During the remote copy process, the OUI will execute the UNIX "tar" command on theremote node (linux3) to extract the files that were archived and copied over. If the date and time on the nodeperforming the install is greater than that of the node it is copying to, the OUI will throw an error from the "tar"command indicating it is attempting to extract files stamped with a time in the future:

Error while copying directory /u01/app/crs with exclude file list 'null' to nodes 'linux3'.[PRKC-1002 : All the submitted commands did not execute successfully]---------------------------------------------linux3: /bin/tar: ./bin/lsnodes: time stamp 2008-02-13 09:21:34 is 735 s in the future /bin/tar: ./bin/olsnodes: time stamp 2008-02-13 09:21:34 is 735 s in the future ...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as awarning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files(using ls -l) on the remote node (the new Oracle RAC node), they will be missing the time field until the timeon the remote server is greater than the timestamp of the file.

Before attempting to add the new node, ensure that all nodes in the cluster are set as closely as possible to thesame date and time. Oracle strongly recommends using the Network Time Protocol feature of most operatingsystems for this purpose, with all nodes using the same reference Network Time Protocol server.

Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manuallysetting the date and time for the nodes in the cluster, ensure that the date and time of the node you areperforming the software installations from (linux1) is less than the new node being added to the cluster(linux3). I generally use a 20 second difference as shown in the following example:

Show the date and time from linux1:

# dateThu Feb 14 00:17:00 EST 2008

Setting the date and time on the new Oracle RAC node linux3:

# date -s "2/14/2008 00:17:20"

The RAC configuration described in this article does not make use of a Network Time Protocol server.

Configure the "hangcheck-timer" Kernel Module


Oracle 9.0.1 and 9.2.0.1 used a userspace watchdog daemon called watchdogd to monitor the health of the cluster and torestart a RAC node in case of a failure. Starting with Oracle 9.2.0.2 (and still available in Oracle10g Release 2), thewatchdog daemon has been deprecated by a Linux kernel module named hangcheck-timer which addresses availabilityand reliability problems much better. The hang-check timer is loaded into the Linux kernel and checks if the system hangs.It will set a timer and check the timer after a certain amount of time. There is a configurable threshold to hang-check that, ifexceeded will reboot the machine. Although the hangcheck-timer module is not required for Oracle Clusterware (ClusterManager) operation, it is highly recommended by Oracle.

The hangcheck-timer.ko Module

The hangcheck-timer module uses a kernel-based timer that periodically checks the system task scheduler tocatch delays in order to determine the health of the system. If the system hangs or pauses, the timer resets thenode. The hangcheck-timer module uses the Time Stamp Counter (TSC) CPU register which is a counterthat is incremented at each clock signal. The TCS offers much more accurate time measurements since thisregister is updated by the hardware automatically.

Much more information about the hangcheck-timer project can be found here.

Installing the hangcheck-timer.ko Module

The hangcheck-timer was normally shipped only by Oracle, however, this module is now included with RedHat Linux AS starting with kernel versions 2.4.9-e.12 and higher. The hangcheck-timer should already beincluded. Use the following to ensure that you have the module included:



# find /lib/modules -name "hangcheck-timer.ko"/lib/modules/2.6.9-55.EL/kernel/drivers/char/hangcheck-timer.ko

In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in the/lib/modules/2.6.9-55.EL/kernel/drivers/char directory.

Configuring and Loading the hangcheck-timer Module

There are two key parameters to the hangcheck-timer module:

hangcheck-tick: This parameter defines the period of time between checks of system health. Thedefault value is 60 seconds; Oracle recommends setting it to 30 seconds.

hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated beforehangcheck-timer resets the RAC node. It defines the margin of error in seconds. The default value is 180seconds; Oracle recommends setting it to 180 seconds.

The two hangcheck-timer module parameters indicate how long a RAC nodemust hang before it will reset the system. A node reset will occur when thefollowing is true:system hang time > (hangcheck_tick + hangcheck_margin)

Configuring Hangcheck Kernel Module Parameters

Each time the hangcheck-timer kernel module is loaded (manually or by Oracle) it needs to knowwhat value to use for each of the two parameters we just discussed: (hangcheck-tick andhangcheck-margin).

These values need to be available after each reboot of the Linux server. To do this, make an entrywith the correct values to the /etc/modprobe.conf file as follows:

# su -# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.conf

Each time the hangcheck-timer kernel module gets loaded, it will use the values defined by theentry I made in the /etc/modprobe.conf file.

Manually Loading the Hangcheck Kernel Module for Testing

Oracle is responsible for loading the hangcheck-timer kernel module when required. It is for thisreason that it is not required to perform a modprobe or insmod of the hangcheck-timer kernelmodule in any of the startup files (i.e. /etc/rc.local).

It is only out of pure habit that I continue to include a modprobe of the hangcheck-timer kernelmodule in the /etc/rc.local file. Someday I will get over it, but realize that it does not hurt toinclude a modprobe of the hangcheck-timer kernel module during startup.

So to keep myself sane and able to sleep at night, I always configure the loading of the hangcheck-timer kernel module on each startup as follows:

# echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local

You don't have to manually load the hangcheck-timer kernel module usingmodprobe or insmod after each reboot. The hangcheck-timer module will beloaded by Oracle (automatically) when needed.

Now, to test the hangcheck-timer kernel module to verify it is picking up the correct parameters wedefined in the /etc/modprobe.conf file, use the modprobe command. Although you could load thehangcheck-timer kernel module by passing it the appropriate parameters (e.g. insmodhangcheck-timer hangcheck_tick=30 hangcheck_margin=180), we want to verify that it is pickingup the options we set in the /etc/modprobe.conf file.

To manually load the hangcheck-timer kernel module and verify it is using the correct valuesdefined in the /etc/modprobe.conf file, run the following command:

# su -# modprobe hangcheck-timer# grep Hangcheck /var/log/messages | tail -2Feb 14 01:22:52 linux3 kernel: Hangcheck: starting hangcheck timer 0.9.0 (tick is 30 seconds, margin is 180 seconds).Feb 14 01:22:52 linux3 kernel: Hangcheck: Using monotonic_clock().

Configure RAC Nodes for Remote Access using SSH

Perform the following configuration procedures on linux1 and the newOracle RAC node!



During the creation of the existing two-node cluster, the installation of Oracle Clusterware and the Oracle Databasesoftware were only performed from one node in the RAC cluster namely from linux1 as the oracle user account. TheOracle Universal Installer (OUI) on that particular node then would use the ssh and scp commands to run remotecommands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The oracle user account onthe node running the OUI (runInstaller) had to be trusted by all other nodes in the RAC cluster. This meant that theoracle user account had to run the secure shell commands (ssh or scp) on the Linux server executing the OUI against allother Linux servers in the cluster without being prompted for a password. The same security requirements hold true for thisarticle. User equivalence will be configured so that the Oracle Clusterware and Oracle Database software will be securelycopied from linux1 to the new Oracle RAC node (linux3) using ssh and scp without being prompted for a password.

As was the case when configuring the existing two-node cluster, this article assumes the Oracle software installation to thenew Oracle RAC node will be performed from linux1. This section provides the methods required for configuring SSH1,an RSA key, and user equivalence for the new Oracle RAC node.

Configuring the Secure Shell

To determine if SSH is installed and running on the new Oracle RAC node, enter the following command:# pgrep sshd3695

If SSH is running, then the response to this command is a list of process ID number(s).

Creating the RSA Keys on the new Oracle RAC Node

The first step in configuring SSH is to create an RSA public/private key pair on the new Oracle RAC node. AnRSA public/private key should already exist on both of the two nodes in the current two-node cluster. Thecommand to do this will create a public and private key for RSA (for a total of two keys per node). The contentof the RSA public keys will then be copied into an authorized key file on linux1 which is then distributed to allother Oracle RAC nodes in the cluster.

Use the following steps to create the RSA key pair from the new Oracle RAC node (linux3);

1. Logon as the "oracle" UNIX user account.# su - oracle

2. If necessary, create the .ssh directory in the "oracle" user's home directory and set the correctpermissions on it:$ mkdir -p ~/.ssh$ chmod 700 ~/.ssh

3. Enter the following command to generate an RSA key pair (public and private key) for the SSH protocol:$ /usr/bin/ssh-keygen -t rsa

At the prompts:Accept the default location for the key files.Enter and confirm a pass phrase. This should be different from the "oracle" UNIX user accountpassword however it is not a requirement.

This command will write the public key to the ~/.ssh/id_rsa.pub file and the private key to the~/.ssh/id_rsa file. Note that you should never distribute the private key to anyone!

Updating and Distributing the "authorized key file" from linux1

Now that the new Oracle RAC node contains a public and private key for RSA, you will need to update theauthorized key file on linux1 to add (append) the new RSA public key from linux3. An authorized key file isnothing more than a single file that contains a copy of everyone's (every node's) RSA public key. Once theauthorized key file contains all of the public keys, it is then distributed to all other nodes in the cluster.

Complete the following steps on linux1 to update and then distribute the authorized key file to all nodes in theOracle RAC cluster.

1. As the "oracle" UNIX user account, verify that an authorized key file currently exists on the node linux1(~/.ssh/authorized_keys). The authorized key file should already exist from the initial installation ofOracle RAC.$ cd ~/.ssh$ ls -l *.pub-rw-r--r-- 1 oracle oinstall 223 Sep 2 01:18 id_rsa.pub

2. In this step, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the content of the~/.ssh/id_rsa.pub public key from the new Oracle RAC node to the authorized key file on linux1. Youwill be prompted for the oracle UNIX user account password for the new Oracle RAC node.

Again, this task will be performed from linux1.

$ ssh linux3 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keysThe authenticity of host 'linux3 (192.168.1.107)' can't be established.RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a.Are you sure you want to continue connecting (yes/no)? yes



Warning: Permanently added 'linux3,192.168.1.107' (RSA) to the list of known hosts.oracle@linux3's password: xxxxx

The first time you use SSH to connect to a node from a particular system, you willsee a message similar to the following:The authenticity of host 'linux3 (192.168.1.107)' can't be established.RSA key fingerprint is f5:38:37:e8:84:4e:bd:6d:6b:25:f7:94:58:e8:b2:7a.Are you sure you want to continue connecting (yes/no)? yes

Enter yes at the prompt to continue. You should not see this message again whenyou connect from this system to the same node.

3. At this point, we have the RSA public key from every node in the cluster (including the new Oracle RACnode) in the authorized key file on linux1. We now need to copy it to all other nodes in the cluster. Usethe scp command to copy the authorized key file to all remaining nodes in the RAC cluster:$ scp ~/.ssh/authorized_keys linux2:.ssh/authorized_keysEnter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxxauthorized_keys 100% 669 0.7KB/s 00:00

$ scp ~/.ssh/authorized_keys linux3:.ssh/authorized_keysoracle@linux3's password: xxxxxauthorized_keys 100% 669 0.7KB/s 00:00

4. Change the permission of the authorized key file on all Oracle RAC nodes in the cluster by logging intothe node and running the following:$ chmod 600 ~/.ssh/authorized_keys

5. At this point, if you use ssh to log in to or run a command on the new Oracle RAC node, you areprompted for the pass phrase that you specified when you created the RSA key. For example, test thefollowing from linux1:$ ssh linux3 hostnameEnter passphrase for key '/home/oracle/.ssh/id_rsa': xxxxxlinux3

If you see any other messages or text, apart from the host name, then the Oracleinstallation can fail. Make any changes required to ensure that only the host nameis displayed when you enter these commands. You should ensure that any part ofa login script(s) that generate any output, or ask any questions, are modified sothat they act only when the shell is an interactive shell.

Enabling SSH User Equivalency for the Current Shell Session

When running the addNode.sh script from linux1 (which runs the OUI), it will need to run the secure shell toolcommands (ssh and scp) on the new Oracle RAC node without being prompted for a pass phrase. Eventhough SSH is now configured on all Oracle RAC nodes in the cluster, using the secure shell tool commandswill still prompt for a pass phrase. Before running the addNode.sh script, you need to enable user equivalencefor the terminal session you plan to run the script from. For the purpose of this article, the addNode.sh script willbe run from linux1.

User equivalence will need to be enabled on any new terminal shell session on linux1 before attempting to runthe addNode.sh script. If you log out and log back in to the node you will be performing the Oracle installationfrom, you must enable user equivalence for the terminal shell session as this is not done by default.

To enable user equivalence for the current terminal shell session, perform the following steps:

1. Logon to the node where you want to run the addNode.sh script from (linux1) as the "oracle" UNIX useraccount.# su - oracle

2. Enter the following commands:$ exec /usr/bin/ssh-agent $SHELL$ /usr/bin/ssh-addEnter passphrase for /home/oracle/.ssh/id_rsa: xxxxxIdentity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa)

At the prompt, enter the pass phrase for each key that you generated.

3. If SSH is configured correctly, you will be able to use the ssh and scp commands without being promptedfor a password or pass phrase from this terminal session:$ ssh linux1 "date;hostname"Fri Feb 22 12:13:57 EST 2008linux1

$ ssh linux2 "date;hostname"Fri Feb 22 12:14:43 EST 2008linux2

$ ssh linux3 "date;hostname"Fri Feb 22 12:13:16 EST 2008linux3



The commands above should display the date set on each Oracle RAC nodealong with its hostname. If any of the nodes prompt for a password or pass phrasethen verify that the ~/.ssh/authorized_keys file on that node contains the correctpublic keys.

Also, if you see any other messages or text, apart from the date and hostname,then the Oracle installation can fail. Make any changes required to ensure that onlythe date is displayed when you enter these commands. You should ensure thatany part of a login script(s) that generate any output, or ask any questions, aremodified so that they act only when the shell is an interactive shell.

4. The Oracle Universal Installer is a GUI interface and requires the use of an X Server. From the terminalsession enabled for user equivalence (the node you will be running the addNode.sh script from), set theenvironment variable DISPLAY to a valid X Windows display:

Bourne, Korn, and Bash shells:

$ DISPLAY=:0$ export DISPLAY

C shell:$ setenv DISPLAY :0

After setting the DISPLAY variable to a valid X Windows display, you should perform another test of thecurrent terminal session to ensure that X11 forwarding is not enabled:$ ssh linux1 hostnamelinux1

$ ssh linux2 hostnamelinux2

$ ssh linux3 hostnamelinux3

5. You must run the addNode.sh script from this terminal session or remember to repeat the steps to enableuser equivalence (steps 2, 3, and 4 from this section) before you start the Oracle Universal Installer froma different terminal session.

For more information on configuring SSH and user equivalence in an Oracle RAC 10g environment, see the section"Configure RAC Nodes for Remote Access using SSH" in the parent article.

All Startup Commands for New Oracle RAC Node

Verify that the following startup commands are included on the newOracle RAC node!

This section will recap all of the parameters, commands, and entries that were covered in previous sections of thisdocument that need to happen on the new Oracle RAC node when the machine is booted. For each of the startup filesbelow, I indicate in blue the entries that should be included in each of the startup files on the new Oracle RAC node.

/etc/modprobe.conf

All parameters and values to be used by kernel modules.

/etc/modprobe.conf

alias eth0 b44alias eth1 e1000alias scsi_hostadapter ata_piixalias usb-controller ehci-hcdalias usb-controller1 uhci-hcdoptions hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

/etc/sysctl.conf

We wanted to adjust the default and maximum send buffer size as well as the default and maximum receivebuffer size for the interconnect. This file also contains those parameters responsible for configuring sharedmemory, semaphores, file handles, and local IP range for use by the Oracle instance.

/etc/sysctl.conf

# Kernel sysctl configuration file for Red Hat Linux## For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and# sysctl.conf(5) for more details.

# Controls IP packet forwardingnet.ipv4.ip_forward = 0



# Controls source route verificationnet.ipv4.conf.default.rp_filter = 1

# Controls the System Request debugging functionality of the kernelkernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.# Useful for debugging multi-threaded applications.kernel.core_uses_pid = 1

# +---------------------------------------------------------+# | ADJUSTING NETWORK SETTINGS |# +---------------------------------------------------------+# | With Oracle 9.2.0.1 and onwards, Oracle now makes use |# | of UDP as the default protocol on Linux for |# | inter-process communication (IPC), such as Cache Fusion |# | and Cluster Manager buffer transfers between instances |# | within the RAC cluster. Oracle strongly suggests to |# | adjust the default and maximum receive buffer size |# | (SO_RCVBUF socket option) to 256 KB, and the default |# | and maximum send buffer size (SO_SNDBUF socket option) |# | to 256 KB. The receive buffers are used by TCP and UDP |# | to hold received data until it is read by the |# | application. The receive buffer cannot overflow because |# | the peer is not allowed to send data beyond the buffer |# | size window. This means that datagrams will be |# | discarded if they don't fit in the socket receive |# | buffer. This could cause the sender to overwhelm the |# | receiver. |# +---------------------------------------------------------+

# +---------------------------------------------------------+# | Default setting in bytes of the socket "receive" buffer |# | which may be set by using the SO_RCVBUF socket option. |# +---------------------------------------------------------+net.core.rmem_default=262144

# +---------------------------------------------------------+# | Maximum setting in bytes of the socket "receive" buffer |# | which may be set by using the SO_RCVBUF socket option. |# +---------------------------------------------------------+net.core.rmem_max=262144

# +---------------------------------------------------------+# | Default setting in bytes of the socket "send" buffer |# | which may be set by using the SO_SNDBUF socket option. |# +---------------------------------------------------------+net.core.wmem_default=262144

# +---------------------------------------------------------+# | Maximum setting in bytes of the socket "send" buffer |# | which may be set by using the SO_SNDBUF socket option. |# +---------------------------------------------------------+net.core.wmem_max=262144

# +---------------------------------------------------------+# | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE |# +---------------------------------------------------------+# | Configure the kernel parameters for all Oracle Linux |# | servers by setting shared memory and semaphores, |# | setting the maximum amount of file handles, and setting |# | the IP local port range. |# +---------------------------------------------------------+

# +---------------------------------------------------------+# | SHARED MEMORY |# +---------------------------------------------------------+kernel.shmmax=2147483648

# +---------------------------------------------------------+# | SEMAPHORES |# | ---------- |# |

Oracle 10g RAC Installation Using the Ocfs and Asmlib and SAN Over Openfiler

Documents