2 Advanced Installation Oracle Grid Infrastructure for a Cluster Preinstallation Tasks

2.4.1 Determining If the Oracle Inventory and Oracle Inventory Group Exists

When you install Oracle software on the system for the first time, OUI creates the oraInst.loc file. This file identifies the name of the Oracle Inventory group (by default, oinstall), and the path of the Oracle Central Inventory directory. An oraInst.loc file has contents similar to the following:

inventory_loc=central_inventory_location
inst_group=group

In the preceding example, central_inventory_location is the location of the Oracle central inventory, and group is the name of the group that has permissions to write to the central inventory (the OINSTALL group privilege).

If you have an existing Oracle central inventory, then ensure that you use the same Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users you intend to use for installation have permissions to write to this directory.

To determine if you have an Oracle central inventory directory (oraInventory) on your system:

Enter the following command:

# more /var/opt/oracle/oraInst.loc

If the oraInst.loc file exists, then the output from this command is similar to the following:

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

In the previous output example:

• The inventory_loc group shows the location of the Oracle Inventory

• The inst_group parameter shows the name of the Oracle Inventory group (in this example, oinstall).

Use the command grep groupname /etc/group to confirm that the group specified as the Oracle Inventory group still exists on the system. For example:

$ grep oinstall /etc/group
oinstall:x:1000:grid,oracle

2.4.2 Creating the Oracle Inventory Group If an Oracle Inventory Does Not Exist

If the oraInst.loc file does not exist, then create the Oracle Inventory group by entering a command similar to the following:

# /usr/sbin/groupadd -g 1000 oinstall

The preceding command creates the oraInventory group oinstall, with the group ID number 1000. Members of the oraInventory group are granted privileges to write to the Oracle central inventory (oraInventory).

By default, if an oraInventory group does not exist, then the installer lists the primary group of the installation owner for the Oracle Grid Infrastructure for a Cluster software as the oraInventory group. Ensure that this group is available as a primary group for all planned Oracle software installation owners.

2.4.3 Creating the Oracle Grid Infrastructure User

You must create a software owner for Oracle Grid Infrastructure in the following circumstances:

• If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system

• If an Oracle software owner user exists, but you want to use a different operating system user, with different group membership, to separate Oracle Grid Infrastructure administrative privileges from Oracle Database administrative privileges.

  In Oracle documentation, a user created to own only Oracle Grid Infrastructure software installations is called the grid user. A user created to own either all Oracle installations, or only Oracle database installations, is called the oracle user.

# id -a oracle

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

2.4.4 Creating the Oracle Base Directory Path

The Oracle base directory for the grid installation owner is the location where diagnostic and administrative logs, and other logs associated with Oracle ASM and Oracle Clusterware are stored.

If you have created a path for the Oracle Clusterware home that is compliant with Oracle Optimal Flexible Architecture (OFA) guidelines for Oracle software paths then you do not need to create an Oracle base directory. When OUI finds an OFA-compliant path, it creates the Oracle base directory in that path.

For OUI to recognize the path as an Oracle software path, it must be in the form u[00-99]/app, and it must be writable by any member of the oraInventory (oinstall) group. The OFA path for the Oracle base is /u01/app/user, where user is the name of the software installation owner.

Oracle recommends that you create an Oracle Grid Infrastructure Grid home and Oracle base homes manually, particularly if you have separate Oracle Grid Infrastructure for a cluster and Oracle Database software owners, so that you can separate log files.

For example:

# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# mkdir -p /u01/app/oracle
# chown grid:oinstall /u01/app/11.2.0/grid
# chown grid:oinstall /u01/app/grid
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/
# chown -R grid:oinstall /u01

2.4.5 Creating Job Role Separation Operating System Privileges Groups and Users

A Job Role Separation privileges configuration of Oracle ASM is a configuration with groups and users that divide administrative access privileges to the Oracle ASM installation from other administrative privileges users and groups associated with other Oracle installations. Administrative privileges access is granted by membership in separate operating system groups, and installation privileges are granted by using different installation owners for each Oracle installation.

If you prefer, you can allocate operating system user privileges so that you can use one administrative user and one group for operating system authentication for all system privileges on the storage and database tiers.

For example, you can designate the oracle user to be the installation owner for all Oracle software, and designate oinstall to be the group whose members are granted all system privileges for Oracle Clusterware, Oracle ASM, and all Oracle Databases on the servers, and all privileges as installation owners. This group must also be the Oracle Inventory group.

Oracle recommends that you use at least two groups: A system privileges group whose members are granted administrative system privileges, and an installation owner group (the oraInventory group) to provide separate installation privileges (the OINSTALL privilege). To simplify using the defaults for Oracle tools such as Cluster Verification Utility, if you do choose to use a single operating system group to grant all system privileges and the right to write to the oraInventory, then that group name should be oinstall.

• Overview of Creating Job Role Separation Groups and Users

• Creating Database Groups and Users with Job Role Separation

# /usr/sbin/groupadd -g 1031 dba

# /usr/sbin/groupadd -g 1032 oper1

# /usr/sbin/groupadd -g 1020 asmadmin

# /usr/sbin/groupadd -g 1022 asmoper

# /usr/sbin/groupadd -g 1021 asmdba

# id -a oracle

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

# /usr/sbin/usermod -g oinstall -G dba,asmdba oracle

2.4.6 Example of Creating Standard Groups, Users, and Paths

The following is an example of how to create the Oracle Inventory group (oinstall), and a single group (dba) as the OSDBA, OSASM and OSDBA for Oracle ASM groups. In addition, it shows how to create the Oracle Grid Infrastructure software owner (grid), and one Oracle Database owner (oracle) with correct group memberships. This example also shows how to configure an Oracle base path compliant with OFA structure with correct permissions:

# groupadd -g 1000 oinstall
# groupadd -g 1031 dba
# useradd -u 1100 -g oinstall -G dba grid
# useradd -u 1101 -g oinstall -G dba oracle
# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# chown -R grid:oinstall /u01
# mkdir /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01/

After running these commands, you have the following groups and users:

• An Oracle central inventory group, or oraInventory group (oinstall). Members who have the central inventory group as their primary group, are granted the OINSTALL permission to write to the oraInventory directory.

• A single system privileges group that is used as the OSASM, OSDBA, OSDBA for ASM, and OSOPER for ASM group (dba), whose members are granted the SYSASM and SYSDBA privilege to administer Oracle Clusterware, Oracle ASM, and Oracle Database, and are granted SYSASM and OSOPER for ASM access to the Oracle ASM storage.

• An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM group as the secondary group, with its Oracle base directory /u01/app/grid.

• An Oracle Database owner (oracle) with the oraInventory group as its primary group, and the OSDBA group as its secondary group, with its Oracle base directory /u01/app/oracle.

• /u01/app owned by grid:oinstall with 775 permissions before installation, and by root after the root.sh script is run during installation. This ownership and permissions enables OUI to create the Oracle Inventory directory, in the path /u01/app/oraInventory.

• /u01 owned by grid:oinstall before installation, and by root after the root.sh script is run during installation.

• /u01/app/11.2.0/grid owned by grid:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.

• /u01/app/grid owned by grid:oinstall with 775 permissions before installation, and 755 permissions after installation.

• /u01/app/oracle owned by oracle:oinstall with 775 permissions.

2.4.7 Example of Creating Role-allocated Groups, Users, and Paths

The following is an example of how to create role-allocated groups and users that is compliant with an Optimal Flexible Architecture (OFA) deployment:

# groupadd -g 1000 oinstall
# groupadd -g 1020 asmadmin
# groupadd -g 1021 asmdba
# groupadd -g 1031 dba1
# groupadd -g 1041 dba2
# groupadd -g 1022 asmoper
# useradd -u 1100 -g oinstall -G asmadmin,asmdba grid
# useradd -u 1101 -g oinstall -G dba1,asmdba oracle1
# useradd -u 1102 -g oinstall -G dba2,asmdba oracle2
# mkdir -p  /u01/app/11.2.0/grid
# mkdir -p /u01/app/grid
# chown -R grid:oinstall /u01
# mkdir -p /u01/app/oracle1
# chown oracle1:oinstall /u01/app/oracle1
# mkdir -p /u01/app/oracle2
# chown oracle2:oinstall /u01/app/oracle2
# chmod -R 775 /u01

After running these commands, you have the following groups and users:

• An Oracle central inventory group, or oraInventory group (oinstall), whose members that have this group as their primary group are granted permissions to write to the oraInventory directory.

• A separate OSASM group (asmadmin), whose members are granted the SYSASM privilege to administer Oracle Clusterware and Oracle ASM.

• A separate OSDBA for ASM group (asmdba), whose members include grid, oracle1 and oracle2, and who are granted access to Oracle ASM.

• A separate OSOPER for ASM group (asmoper), whose members are granted limited Oracle ASM administrator privileges, including the permissions to start and stop the Oracle ASM instance.

• An Oracle grid installation for a cluster owner (grid), with the oraInventory group as its primary group, and with the OSASM (asmadmin), OSDBA for ASM (asmdba) group as a secondary group.

• Two separate OSDBA groups for two different databases (dba1 and dba2) to establish separate SYSDBA privileges for each database.

• Two Oracle Database software owners (oracle1 and oracle2), to divide ownership of the Oracle database binaries, with the OraInventory group as their primary group, and the OSDBA group for their database (dba1 or dba2) and the OSDBA for ASM group (asmdba) as their secondary groups.

• An OFA-compliant mount point /u01 owned by grid:oinstall before installation.

• An Oracle base for the grid installation owner /u01/app/grid owned by grid:oinstall with 775 permissions, and changed during the installation process to 755 permissions.

• An Oracle base /u01/app/oracle1 owned by oracle1:oinstall with 775 permissions.

• An Oracle base /u01/app/oracle 2 owned by oracle2:oinstall with 775 permissions.

• A Grid home /u01/app/11.2.0/grid owned by grid:oinstall with 775 (drwxdrwxr-x) permissions. These permissions are required for installation, and are changed during the installation process to root:oinstall with 755 permissions (drwxr-xr-x).

• /u01/app/oraInventory. This path remains owned by grid:oinstall, to enable other Oracle software owners to write to the central inventory.

2.6.1 Network Hardware Requirements

The following is a list of requirements for network configuration:

• Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).

  With Redundant Interconnect Usage, you can identify multiple interfaces to use for the cluster private network, without the need of using bonding or other technologies. This functionality is available starting with Oracle Database 11g Release 2 (11.2.0.2).

  When you define multiple interfaces, Oracle Clusterware creates from one to four highly available IP (HAIP) addresses. Oracle RAC and Oracle ASM instances use these interface addresses to ensure highly available, load-balanced interface communication between nodes. The installer enables Redundant Interconnect Usage to provide a high availability private network.

  See Also:

  Oracle Clusterware Administration and Deployment Guide for more information about using OIFCFG to modify interfaces

  By default, Oracle Grid Infrastructure software uses all of the HAIP addresses for private network communication, providing load-balancing across the set of interfaces you identify for the private network. If a private interconnect interface fails or become non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces.

  Note:

  If you define more than four interfaces as private network interfaces, be aware that Oracle Clusterware activates only four of the interfaces at a time. However, if one of the four active interfaces fails, then Oracle Clusterware transitions the HAIP addresses configured to the failed interface to one of the reserve interfaces in the defined set of private interfaces.

  Note:

  If you are installing Oracle Clusterware on Oracle Solaris Cluster, then you should select the Oracle Solaris Cluster virtual network interface clprivnet0 as the clusterware private network address.

  On Oracle Solaris, if you use IP network multipathing (IPMP) to aggregate multiple interfaces for the public or the private networks, then during installation of Oracle Grid Infrastructure, ensure you identify all interface names aggregated into an IPMP group as interfaces that should be used for the public or private network.

  When you upgrade a node to Oracle Grid Infrastructure 11g release 2 (11.2.0.2) and later, the upgraded system uses your existing network classifications. After you complete the upgrade, you can enable Redundant Interconnect Usage by selecting multiple interfaces for the private network with OIFCFG.

  Oracle recommends that you use the Redundant Interconnect Usage feature to make use of multiple interfaces for the private network. However, you can also use third-party technologies to provide redundancy for the private network.

• If you install Oracle Clusterware using OUI, then the public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes. This restriction does not apply if you use cloning, either to create a new cluster, or to add nodes to an existing cluster.

  For example: With a two-node cluster, you cannot configure network adapters on node1 with eth0 as the public interface, but on node2 have eth1 as the public interface. Public interface names must be the same, so you must configure eth0 as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2.

  See Also:

  Oracle Clusterware Administration and Deployment Guide for information about how to add nodes using cloning

• For the public network, each network adapter must support TCP/IP.

• For the private network, the interface must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (minimum requirement 1 Gigabit Ethernet).

  Note:

  UDP is the default interface protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. You must use a switch for the interconnect. Oracle recommends that you use a dedicated switch.

  Oracle does not support token-rings or crossover cables for the interconnect.

• Each node's private interface for interconnects must be on the same subnet, and those subnets must connect to every node of the cluster. For example, if the private interfaces have a subnet mask of 255.255.255.0, then your private network is in the range 192.168.0.0--192.168.0.255, and your private addresses must be in the range of 192.168.0.[0-255]. If the private interfaces have a subnet mask of 255.255.0.0, then your private addresses can be in the range of 192.168.[0-255].[0-255].

• For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test if an interconnect interface is reachable using ping.

2.6.2 IP Address Requirements

Before starting the installation, you must have at least two interfaces configured on each node: One for the private IP address and one for the public IP address.

You can configure IP addresses with one of the following options:

• Dynamic IP address assignment using Oracle Grid Naming Service (GNS). If you select this option, then network administrators assign static IP address for the physical host name and dynamically allocated IPs for the Oracle Clusterware managed VIP addresses. In this case, IP addresses for the VIPs are assigned by a DHCP and resolved using a multicast domain name server configured as part of Oracle Clusterware within the cluster. If you plan to use GNS, then you must have the following:

  • A DHCP service running on the public network for the cluster

  • Enough addresses on the DHCP to provide 1 IP address for each node's virtual IP, and 3 IP addresses for the cluster used by the Single Client Access Name (SCAN) for the cluster

• Static IP address assignment. If you select this option, then network administrators assign a fixed IP address for each physical host name in the cluster and for IPs for the Oracle Clusterware managed VIPs. In addition, domain name server (DNS) based static name resolution is used for each node. Selecting this option requires that you request network administration updates when you modify the cluster.

root@node1]$ nslookup mycluster-scan
Server:         dns.example.com
Address:        192.0.2.001
 
Name:   mycluster-scan.example.com
Address: 192.0.2.201
Name:   mycluster-scan.example.com
Address: 192.0.2.202
Name:   mycluster-scan.example.com
Address: 192.0.2.203

2.6.3 Broadcast Requirements for Networks Used by Oracle Grid Infrastructure

Broadcast communications (ARP and UDP) must work properly across all the public and private interfaces configured for use by Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases.

The broadcast must work across any configured VLANs as used by the public or private interfaces.

2.6.4 Multicast Requirements for Networks Used by Oracle Grid Infrastructure

With Oracle Grid Infrastructure release 2 (11.2), on each cluster member node, the Oracle mDNS daemon uses multicasting on all interfaces to communicate with other nodes in the cluster.

With Oracle Grid Infrastructure release 2 patchset 1 (11.2.0.2) and later releases, multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster:

• Across the broadcast domain as defined for the private interconnect

• On the IP address subnet ranges 224.0.0.0/24 and 230.0.1.0/24

You do not need to enable multicast communications across routers.

2.6.5 DNS Configuration for Domain Delegation to Grid Naming Service

If you plan to use GNS, then before Oracle Grid Infrastructure installation, you must configure your domain name server (DNS) to send to GNS name resolution requests for the subdomain GNS serves, which are the cluster member nodes. The following is an overview of what needs to be done for domain delegation. Your actual procedure may be different from this example.

Configure the DNS to send GNS name resolution requests using delegation:

1. In the DNS, create an entry for the GNS virtual IP address, where the address uses the form gns-server.CLUSTERNAME.DOMAINNAME. For example, where the cluster name is mycluster, and the domain name is example.com, and the IP address is 192.0.2.1, create an entry similar to the following:

   mycluster-gns.example.com  A  192.0.2.1
   

   The address you provide must be routable.

2. Set up forwarding of the GNS subdomain to the GNS virtual IP address, so that GNS resolves addresses to the GNS subdomain. To do this, create a BIND configuration entry similar to the following for the delegated domain, where cluster01.example.com is the subdomain you want to delegate:

   cluster01.example.com  NS  mycluster-gns.example.com
   

3. When using GNS, you must configure resolve.conf on the nodes in the cluster (or the file on your system that provides resolution information) to contain name server entries that are resolvable to corporate DNS servers. The total timeout period configured—a combination of options attempts (retries) and options timeout (exponential backoff)—should be less than 30 seconds. For example, where xxx.xxx.xxx.42 and xxx.xxx.xxx.15 are valid name server addresses in your network, provide an entry similar to the following in /etc/resolv.conf:

   options attempts: 2
   options timeout: 1
   
   search cluster01.example.com example.com
   nameserver xxx.xxx.xxx.42
   nameserver xxx.xxx.xxx.15
   

   /etc/nsswitch.conf controls name service lookup order. In some system configurations, the Network Information System (NIS) can cause problems with Oracle SCAN address resolution. Oracle recommends that you place the nis entry at the end of the search list. For example:

   /etc/nsswitch.conf
        hosts:    files   dns   nis
   

   Note:

   Be aware that use of NIS is a frequent source of problems when doing cable pull tests, as host name and username resolution can fail.

2.6.6 Grid Naming Service Configuration Example

If you use GNS, then you need to specify a static IP address for the GNS VIP address, and delegate a subdomain to be delegated to that static GNS IP address.

As nodes are added to the cluster, your organization's DHCP server can provide addresses for these nodes dynamically. These addresses are then registered automatically in GNS, and GNS provides resolution within the subdomain to cluster node addresses registered with GNS.

Because allocation and configuration of addresses is performed automatically with GNS, no further configuration is required. Oracle Clusterware provides dynamic network configuration as nodes are added to or removed from the cluster. The following example is provided only for information.

With a two node cluster where you have defined the GNS VIP, after installation you might have a configuration similar to the following for a two-node cluster, where the cluster name is mycluster, the GNS parent domain is example.com, the subdomain is grid.example.com, 192.0.2 in the IP addresses represent the cluster public IP address network, and 192.168.0 represents the private IP address subnet:

^Footnote 1 Node host names may resolve to multiple addresses, including VIP addresses currently running on that host.

2.6.7 Manual IP Address Configuration Example

If you choose not to use GNS, then before installation you must configure public, virtual, and private IP addresses. Also, check that the default gateway can be accessed by a ping command. To find the default gateway, use the route command, as described in your operating system's help utility.

For example, with a two node cluster where each node has one public and one private interface, and you have defined a SCAN domain address to resolve on your DNS to one of three IP addresses, you might have the configuration shown in the following table for your network interfaces:

^Footnote 1 Node host names may resolve to multiple addresses.

You do not need to provide a private name for the interconnect. If you want name resolution for the interconnect, then you can configure private IP names in the hosts file or the DNS. However, Oracle Clusterware assigns interconnect addresses on the interface defined during installation as the private interface (eth1, for example), and to the subnet used for the private subnet.

The addresses to which the SCAN resolves are assigned by Oracle Clusterware, so they are not fixed to a particular node. To enable VIP failover, the configuration shown in the preceding table defines the SCAN addresses and the public and VIP addresses of both nodes on the same subnet, 192.0.2.

2.6.8 Network Interface Configuration Options

The precise configuration you choose for your network depends on the size and use of the cluster you want to configure, and the level of availability you require.

If certified Network-attached Storage (NAS) is used for Oracle RAC and this storage is connected through Ethernet-based networks, then you must have a third network interface for NAS I/O. Failing to provide three separate interfaces in this case can cause performance and stability problems under load.

2.6.9 Checking the Run Level and Name Service Cache Daemon

To allow Oracle Clusterware to better tolerate network failures with NAS devices or NFS mounts, enable the Name Service Cache Daemon (nscd). The nscd provides a caching mechanism for the most common name service requests. It is automatically started when the system starts up in a multi-user state. Oracle software requires that the server is started with multiuser run level (3), which is the default for Oracle Solaris.

To check to see if the server is set to 3, enter the command who -r. For example:

# who -r 
.       run-level 3  Jan 4 14:04     3      0  S 

Refer to your operating system documentation if you need to change the run level.

To check to see if the name service cache daemon is running, enter the following command:

# svcs svc:/sysstem/name-service-cache
STATE STIME FMRI
online Aug_28 svc:/system/name-service-cache:default

Alternatively, enter the command ps -aef |grep nscd.

2.6.10 Checking Oracle Solaris Service Management Facility Status

Use svcs svc commands to check to see if Service Management Facility (SMF) is enabled on your system, and the multi-user and multi-user-server services are online:

# svcs svc:/milestone/multi-user
STATE STIME FMRI
online Aug_28 svc:/milestone/multi-user:default
# svcs svc:/milestone/multi-user-server
STATE STIME FMRI
online Aug_28 svc:/milestone/multi-user-server:default

2.7.1 Software Requirements List for Oracle Solaris (SPARC 64-Bit) Platforms

SUNWarc
SUNWbtool
SUNWcsl
SUNWhea
SUNWi1cs (ISO8859-1)
SUNWi15cs (ISO8859-15)
SUNWi1of
SUNWlibC
SUNWlibm
SUNWlibms
SUNWsprot
SUNWtoo
SUNWxwfnt
119963-14: Sun OS 5.10: Shared Library Patch for C++
120753-06: SunOS 5.10: Microtasking libraries (libmtsk) patch
139574-03: SunOS 5.10
141444-02
141414-02
141414-09 (11.2.0.2 or later)

http://java.sun.com/j2se/1.4.2/font-requirements.html

125555-03
139555-08
140796-01
140899-01
141016-01
141414-10
141736-05

137585-05  or later (IPMItool patch)
137594-02  or later (BMC driver patch)

125508-08
125514-05
125992-04
126047-11
126095-05
126106-33

2.7.2 Software Requirements List for Oracle Solaris (x86 64-Bit) Platforms

SUNWarc
SUNWbtool
SUNWcsl
SUNWhea
SUNWlibC
SUNWlibm
SUNWlibms
SUNWsprot
SUNWtoo
SUNWi1of
SUNWi1cs (ISO8859-1)
SUNWi15cs (ISO8859-15)
SUNWxwfnt
119961-05: SunOS 5.10_x86: Assembler
119964-14: SunOS 5.10_x86 Shared library patch for C++_x86
120754-06: SunOS 5.10_x86 libmtsk
137104-02
139575-03
139556-08
141415-04
141445-09 (11.2.0.2)

http://java.sun.com/j2se/1.4.2/font-requirements.html

139556-08
140797-01
140900-01
141017-01
141415-10
141737-05

125509-10
125515-05
125993-04
126048-11
126096-04
126096-05
126107-33
137104-02

2.14.1 Requirements for Enabling IPMI

You must have the following hardware and software configured to enable cluster nodes to be managed with IPMI:

• Each cluster member node requires a Baseboard Management Controller (BMC) running firmware compatible with IPMI version 1.5 or greater, which supports IPMI over LANs, and configured for remote control using LAN.

• The cluster requires a management network for IPMI. This can be a shared network, but Oracle recommends that you configure a dedicated network.

• Each cluster member node's port used by BMC must be connected to the IPMI management network.

• Each cluster member must be connected to the management network.

• Some server platforms put their network interfaces into a power saving mode when they are powered off. In this case, they may operate only at a lower link speed (for example, 100 MB, instead of 1 GB). For these platforms, the network switch port to which the BMC is connected must be able to auto-negotiate down to the lower speed, or IPMI will not function properly.

• Install and configure IPMI firmware patches as described in Section 2.14.1.1, "IPMI Firmware Patches."

2.14.2 Configuring the IPMI Management Network

On Oracle Solaris platforms, the BMC shares configuration information with the Integrated Lights Out Manager service processor (ILOM). For Oracle Clusterware, you must configure the ILOM/BMC for static IP addresses. Configuring the BMC with dynamic addresses (DHCP) is not supported on Oracle Solaris.

2.14.3 Configuring the BMC

On each node, complete the following steps to configure the BMC to support IPMI-based node fencing:

• Enable IPMI over LAN, so that the BMC can be controlled over the management network.

• Configure a static IP address for the BMC.

• Establish an administrator user account and password for the BMC

• Configure the BMC for VLAN tags, if you will use the BMC on a tagged VLAN.

The configuration tool you use does not matter, but these conditions must be met for the BMC to function properly.

Refer to the documentation for the configuration option you select for details about configuring the BMC.

2.16.1 Environment Requirements for Oracle Grid Infrastructure Software Owner

You must make the following changes to configure the Oracle Grid Infrastructure software owner environment:

• Set the installation software owner user (grid, oracle) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

• Set ulimit settings for file descriptors and processes for the installation software owner (grid, oracle)

• Set the software owner's environment variable DISPLAY environment variables in preparation for the Oracle Grid Infrastructure installation

2.16.2 Procedure for Configuring Oracle Software Owner Environments

To set the Oracle software owners' environments, follow these steps, for each software owner (grid, oracle):

1. Start a new terminal session; for example, start an X terminal (xterm).

2. Enter the following command to ensure that X Window applications can display on this system:

   $ xhost + hostname
   

   The hostname is the name of the local host.

3. If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.

4. If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the grid user:

   $ su - grid
   

5. To determine the default shell for the user, enter the following command:

   $ echo $SHELL
   

   Caution:

   Use shell programs supported by your operating system vendor. If you use a shell program that is not supported by your operating system, then you can encounter errors during installation.

6. Open the user's shell startup file in any text editor:

   • Bash shell (bash):

     $ vi .bash_profile
     

   • Bourne shell (sh) or Korn shell (ksh):

     $ vi .profile
     

   • C shell (csh or tcsh):

     % vi .login
     

7. Enter or edit the following line, specifying a value of 022 for the default file mode creation mask:

   umask 022
   

8. If the ORACLE_SID, ORACLE_HOME, or ORACLE_BASE environment variables are set in the file, then remove these lines from the file.

9. Save the file, and exit from the text editor.

10. To run the shell startup script, enter one of the following commands:

    • Bash shell:

      $ . ./.bash_profile
      

    • Bourne, Bash, or Korn shell:

      $ . ./.profile
      

    • C shell:

      % source ./.login
      

11. If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:

    • Bourne, Bash, or Korn shell:

      $ DISPLAY=local_host:0.0 ; export DISPLAY
      

    • C shell:

      % setenv DISPLAY local_host:0.0
      

    In this example, local_host is the host name or IP address of the system (your workstation, or another client) on which you want to display the installer.

12. If you determined that the /tmp directory has less than 1 GB of free space, then identify a file system with at least 1 GB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system:

    Note:

    You cannot use a shared file system as the location of the temporary file directory (typically /tmp) for Oracle RAC installation. If you place /tmp on a shared file system, then the installation fails.

    1. Use the df -h command to identify a suitable file system with sufficient free space.

    2. If necessary, enter commands similar to the following to create a temporary directory on the file system that you identified, and set the appropriate permissions on the directory:

       $ su - root
       # mkdir /mount_point/tmp
       # chmod 775 /mount_point/tmp
       # exit
       

    3. Enter commands similar to the following to set the TEMP and TMPDIR environment variables:

       • Bourne, Bash, or Korn shell:

         $ TEMP=/mount_point/tmp
         $ TMPDIR=/mount_point/tmp
         $ export TEMP TMPDIR
         

       • C shell:

         % setenv TEMP /mount_point/tmp
         % setenv TMPDIR /mount_point/tmp
         

13. To verify that the environment has been set correctly, enter the following commands:

    $ umask
    $ env | more
    

    Verify that the umask command displays a value of 22, 022, or 0022 and that the environment variables you set in this section have the correct values.

2.16.3 Configuring Shell Limits

Oracle recommends that you set shell limits and system configuration parameters as described in this section.

The ulimit settings determine process memory related resource limits. Verify that the shell limits displayed in the following table are set to the values shown:

To display the current value specified for these shell limits enter the following commands:

ulimit -t
ulimit -f
ulimit -d
ulimit -s
ulimit -n
ulimit -v

2.16.4 Setting Display and X11 Forwarding Configuration

If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:

Bourne, Korn, and Bash shells

$ export DISPLAY=hostname:0

C shell:

$ setenv DISPLAY hostname:0

For example, if you are using the Bash shell, and if your host name is node1, then enter the following command:

$ export DISPLAY=node1:0

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:

1. Using any text editor, edit or create the software installation owner's ~/.ssh/config file.

2. Make sure that the ForwardX11 attribute is set to no. For example:

   Host *
         ForwardX11 no
   

2.16.5 Preventing Installation Errors Caused by Terminal Output Commands

During an Oracle Grid Infrastructure installation, OUI uses SSH to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc or .cshrc) will cause makefile and other installation errors if they contain stty commands.

To avoid this problem, you must modify these files in each Oracle installation owner user home directory to suppress all output on STDOUT or STDERR (for example, stty, xtitle, and other such commands) as in the following examples:

• Bourne, Bash, or Korn shell:

  if [ -t 0 ]; then
     stty intr ^C
  fi
  

• C shell:

  test -t 0
  if ($status == 0) then
     stty intr ^C
  endif
  

  Note:

  When SSH is not available, the Installer uses the rsh and rcp commands instead of ssh and scp.

  If there are hidden files that contain stty commands that are loaded by the remote shell, then OUI indicates an error and stops the installation.