3 Configuring Storage for Grid Infrastructure for a Cluster and Oracle Real Application Clusters (Oracle RAC)

3.1.2.1 General Storage Considerations for Oracle Clusterware

Oracle Clusterware voting disks are used to monitor cluster node status, and Oracle Cluster Registry (OCR) files contain configuration information about the cluster. You can place voting disks and OCR files either in an Oracle ASM disk group, or on a cluster file system or shared network file system. Storage must be shared; any node that does not have access to an absolute majority of voting disks (more than half) will be restarted.

3.1.2.2 General Storage Considerations for Oracle RAC

Use the following guidelines when choosing the storage options to use for each file type:

• You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.

• Oracle recommends that you choose Oracle ASM as the storage option for database and recovery files.

• For Standard Edition Oracle RAC installations, Oracle ASM is the only supported storage option for database or recovery files.

• If you intend to use Oracle ASM with Oracle RAC, and you are configuring a new Oracle ASM instance, then your system must meet the following conditions:

  • All nodes on the cluster have Oracle Clusterware and Oracle ASM 11g release 2 (11.2) installed as part of an Oracle Grid Infrastructure for a cluster installation.

  • Any existing Oracle ASM instance on any node in the cluster is shut down.

• Raw or block devices are supported only when upgrading an existing installation using the partitions already configured. On new installations, using raw or block device partitions is not supported by Oracle Automatic Storage Management Configuration Assistant (ASMCA) or Oracle Universal Installer (OUI), but is supported by the software if you perform manual configuration.

  See Also:

  Oracle Database Upgrade Guide for information about how to prepare for upgrading an existing database

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy.

3.2.3.1 About Direct NFS Storage

With Oracle Database 11g release 2 (11.2), instead of using the operating system kernel NFS client, you can configure Oracle Database to access NFS V3 servers directly using an Oracle internal Direct NFS client.

To enable Oracle Database to use Direct NFS, the NFS file systems must be mounted and available over regular NFS mounts before you start installation. Direct NFS manages settings after installation. You should still set the kernel mount options as a backup, but for normal operation, Direct NFS will manage NFS mounts.

Refer to your vendor documentation to complete NFS configuration and mounting.

Some NFS file servers require NFS clients to connect using reserved ports. If your filer is running with reserved port checking, then you must disable it for Direct NFS to operate. To disable reserved port checking, consult your NFS file server documentation.

3.2.3.2 Using the Oranfstab File with Direct NFS

If you use Direct NFS, then you can choose to use a new file specific for Oracle data file management, oranfstab, to specify additional options specific for Oracle Database to Direct NFS. For example, you can use oranfstab to specify additional paths for a mount point. You can add the oranfstab file either to /etc or to $ORACLE_HOME/dbs.

With shared Oracle homes, when the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database. In this case, all nodes running an Oracle RAC database use the same $ORACLE_HOME/dbs/oranfstab file. In non-shared Oracle RAC installs, oranfstab must be replicated on all nodes.

When the oranfstab file is placed in /etc, then it is globally available to all Oracle databases, and can contain mount points used by all Oracle databases running on nodes in the cluster, including standalone databases. However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file /etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the /etc/fstab file.

In all cases, mount points must be mounted by the kernel NFS system, even when they are being served using Direct NFS.

3.2.3.3 Mounting NFS Storage Devices with Direct NFS

Direct NFS determines mount point settings to NFS storage devices based on the configurations in /etc/mnttab, which are changed with configuring the /etc/fstab file.

Direct NFS searches for mount entries in the following order and uses the first matching entry it finds:

1. $ORACLE_HOME/dbs/oranfstab

2. /etc/oranfstab

3. /etc/mnttab

Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, complete the following steps:

1. Change the directory to $ORACLE_HOME/rdbms/lib.

2. Enter the following command:

   make -f ins_rdbms.mk dnfs_on
   

If Oracle Database uses Direct NFS mount points configured using oranfstab, then it first verifies kernel NFS mounts by cross-checking entries in oranfstab with operating system NFS mount points. If a mismatch exists, then Direct NFS logs an informational message, and does not operate.

Section 3.1.5, "Supported Storage Options" lists the file types that are supported by Direct NFS.

If Oracle Database cannot open an NFS server using Direct NFS, then Oracle Database uses the platform operating system kernel NFS client. In this case, the kernel NFS mount options must be set up as defined in Section 3.2.7, "Checking NFS Mount and Buffer Size Parameters for Oracle RAC." Additionally, an informational message is logged into the Oracle alert and trace files indicating that Direct NFS could not be established. The Oracle files resident on the NFS server that are served by the Direct NFS Client are also accessible through the operating system kernel NFS client.

3.2.3.4 Specifying Network Paths with the Oranfstab File

Direct NFS can use up to four network paths defined in the oranfstab file for an NFS server. The Direct NFS client performs load balancing across all specified paths. If a specified path fails, then Direct NFS reissues I/O commands over any remaining paths.

Use the following SQL*Plus views for managing Direct NFS in a cluster environment:

• gv$dnfs_servers: Shows a table of servers accessed using Direct NFS.

• gv$dnfs_files: Shows a table of files currently open using Direct NFS.

• gv$dnfs_channels: Shows a table of open network paths (or channels) to servers for which Direct NFS is providing files.

• gv$dnfs_stats: Shows a table of performance statistics for Direct NFS.

3.3.1.1 Identifying Storage Requirements for Oracle ASM

To identify the storage requirements for using Oracle ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use Oracle ASM for Oracle Clusterware files (OCR and voting disks), Oracle Database files, recovery files, or all files except for Oracle Clusterware or Oracle Database binaries. Oracle Database files include data files, control files, redo log files, the server parameter file, and the password file.

   Note:

   You do not have to use the same storage mechanism for Oracle Clusterware, Oracle Database files and recovery files. You can use a shared file system for one file type and Oracle ASM for the other.

   If you choose to enable automated backups and you do not have a shared file system available, then you must choose Oracle ASM for recovery file storage.

   If you enable automated backups during the installation, then you can select Oracle ASM as the storage mechanism for recovery files by specifying an Oracle Automatic Storage Management disk group for the Fast Recovery Area. If you select a noninteractive installation mode, then by default it creates one disk group and stores the OCR and voting disk files there. If you want to have any other disk groups for use in a subsequent database install, then you can choose interactive mode, or run ASMCA (or a command line tool) to create the appropriate disk groups before starting the database install.

2. Choose the Oracle ASM redundancy level to use for the Oracle ASM disk group.

   The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group and determines the number of disks and amount of free disk space that you require, as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     For Oracle Clusterware files, External redundancy disk groups provide 1 voting disk file, and 1 OCR, with no copies. You must use an external technology to provide mirroring for high availability.

     Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.

   • Normal redundancy

     In a normal redundancy disk group, to increase performance and reliability, Oracle ASM by default uses two-way mirroring. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For Oracle Clusterware files, Normal redundancy disk groups provide 3 voting disk files, 1 OCR and 2 copies (one primary and one secondary mirror). With normal redundancy, the cluster can survive the loss of one failure group.

     For most installations, Oracle recommends that you select normal redundancy.

   • High redundancy

     In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     For Oracle Clusterware files, High redundancy disk groups provide 5 voting disk files, 1 OCR and 3 copies (one primary and two secondary mirrors). With high redundancy, the cluster can survive the loss of two failure groups.

     While high redundancy disk groups do provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to select high redundancy disk groups.

3. Determine the total amount of disk space that you require for Oracle Clusterware files, and for the database files and recovery files.

   Use Table 3-4 and Table 3-5 to determine the minimum number of disks and the minimum disk space requirements for installing Oracle Clusterware files, and installing the starter database, where you have voting disks in a separate disk group:

   Table 3-4 Total Oracle Clusterware Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Oracle Cluster Registry (OCR) Files	Voting Disk Files	Both File Types
   External	1	300 MB	300 MB	600 MB
   Normal	3	600 MB	900 MB	1.5 GBFoot 1
   High	5	840 MB	1.4 GB	4 GB

   
   

   ^Footnote 1 If you create a disk group during installation, then it must be at least 2 GB.

   Note:

   If the voting disk files are in a disk group, be aware that disk groups with Oracle Clusterware files (OCR and voting disk files) have a higher minimum number of failure groups than other disk groups.

   If you create a disk group as part of the installation in order to install the OCR and voting disk files, then the installer requires that you create these files on a disk group with at least 2 GB of available space.

   A quorum failure group is a special type of failure group and disks in these failure groups do not contain user data. A quorum failure group is not considered when determining redundancy requirements in respect to storing user data. However, a quorum failure group counts when mounting a disk group.

   Table 3-5 Total Oracle Database Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Database Files	Recovery Files	Both File Types
   External	1	1.5 GB	3 GB	4.5 GB
   Normal	2	3 GB	6 GB	9 GB
   High	3	4.5 GB	9 GB	13.5 GB

   
   

4. For Oracle Clusterware installations, you must also add additional disk space for the Oracle ASM metadata. You can use the following formula to calculate the additional disk space requirements (in MB) for OCR and voting disk files, and the Oracle ASM metadata:

   total = [2 * ausize * disks] + [redundancy * (ausize * (nodes * (clients + 1) + 30) + (64 * nodes) + 533)]

   Where:

   • redundancy = Number of mirrors: external = 1, normal = 2, high = 3.

   • ausize = Metadata AU size in megabytes.

   • nodes = Number of nodes in cluster.

   • clients - Number of database instances for each node.

   • disks - Number of disks in disk group.

   For example, for a four-node Oracle RAC installation, using three disks in a normal redundancy disk group, you require an additional X MB of space:

   [2 * 1 * 3] + [2 * (1 * (4 * (4 + 1)+ 30)+ (64 * 4)+ 533)] = 1684 MB

   To ensure high availability of Oracle Clusterware files on Oracle ASM, you must have at least 2 GB of disk space for Oracle Clusterware files in three separate failure groups, with at least three physical disks. Each disk must have at least 1 GB of capacity to ensure that there is sufficient space to create Oracle Clusterware files.

5. Optionally, identify failure groups for the Oracle ASM disk group devices.

   If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   Note:

   Define custom failure groups after installation, using the GUI tool ASMCA, the command line tool asmcmd, or SQL commands.

   If you define custom failure groups, then for failure groups containing database files only, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

   For failure groups containing database files and clusterware files, including voting disks, you must specify a minimum of three failure groups for normal redundancy disk groups, and five failure groups for high redundancy disk groups.

   Disk groups containing voting files must have at least 3 failure groups for normal redundancy or at least 5 failure groups for high redundancy. Otherwise, the minimum is 2 and 3 respectively. The minimum number of failure groups applies whether or not they are custom failure groups.

6. If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify multiple partitions on a single physical disk as a disk group device. Oracle ASM expects each disk group device to be on a separate physical disk.

   • Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.

     Oracle recommends that if you choose to use a logical volume manager, then use the logical volume manager to represent a single LUN without striping or mirroring, so that you can minimize the impact of the additional storage layer.

3.3.1.2 Creating Files on a NAS Device for Use with Oracle ASM

If you have a certified NAS storage device, then you can create zero-padded files in an NFS mounted directory and use those files as disk devices in an Oracle ASM disk group.

To create these files, follow these steps:

1. If necessary, create an exported directory for the disk group files on the NAS device.

   Refer to the NAS device documentation for more information about completing this step.

2. Switch user to root.

3. Create a mount point directory on the local system.

4. To ensure that the NFS file system is mounted when the system restarts, add an entry for the file system in the mount file /etc/fstab.

   See Also:

   My Oracle Support note 359515.1 for updated NAS mount option information, available at the following URL:

   https://support.oracle.com
   

   For more information about editing the mount file for the operating system, refer to the man pages. For more information about recommended mount options, refer to the section Section 3.2.7, "Checking NFS Mount and Buffer Size Parameters for Oracle RAC."

5. Mount the NFS file system on the local system

6. Choose a name for the disk group to create. For example: sales1.

7. Create a directory for the files on the NFS file system, using the disk group name as the directory name.

8. Use commands similar to the following to create the required number of zero-padded files in this directory:

   # dd if=/dev/zero of=/mnt/nfsdg/disk1 bs=1024k count=1000
   

   This example creates 1 GB files on the NFS file system. You must create one, two, or three files respectively to create an external, normal, or high redundancy disk group.

9. Enter commands similar to the following to change the owner, group, and permissions on the directory and files that you created, where the installation owner is grid, and the OSASM group is asmadmin:

   # chown -R grid:asmadmin /mnt/nfsdg
   # chmod -R 660 /mnt/nfsdg
   

10. If you plan to install Oracle RAC or a standalone Oracle Database, then during installation, edit the Oracle ASM disk discovery string to specify a regular expression that matches the file names you created. For example:

    /mnt/nfsdg/sales1/
    

3.3.1.3 Using an Existing Oracle ASM Disk Group

Select from the following choices to store either database or recovery files in an existing Oracle ASM disk group, depending on installation method:

• If you select an installation method that runs Database Configuration Assistant in interactive mode, then you can decide whether you want to create a disk group, or to use an existing one.

  The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

• If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine if an existing Oracle ASM disk group exists, or to determine if there is sufficient disk space in a disk group, you can use the Oracle ASM command line tool (asmcmd), Oracle Enterprise Manager Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine if an Oracle ASM instance is configured on the system:

   $ more /etc/oratab
   

   If an Oracle ASM instance is configured on the system, then the oratab file should contain a line similar to the following:

   +ASM2:oracle_home_path
   

   In this example, +ASM2 is the system identifier (SID) of the Oracle ASM instance, with the node number appended, and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an Oracle ASM instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the Oracle ASM instance.

3. Connect to the Oracle ASM instance and start the instance if necessary:

   $ $ORACLE_HOME/bin/asmcmd
   ASMCMD> startup
   

4. Enter one of the following commands to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   ASMCMD> lsdb
   

   or:

   $ORACLE_HOME/bin/asmcmd -p lsdg
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   Note:

   If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

3.3.1.4 Configuring Disk Devices for Oracle ASM

To configure disks for use with Oracle ASM on HP-UX, follow these steps:

1. If necessary, install the shared disks that you intend to use for the Oracle ASM disk group.

2. To make sure that the disks are available, enter the following command on every node:

   # /usr/sbin/ioscan -fun -C disk
   

   The output from this command is similar to the following:

   Class  I  H/W Path    Driver S/W State   H/W Type     Description
   ==========================================================================
   disk    0  0/0/1/0.6.0 sdisk  CLAIMED     DEVICE       HP   DVD-ROM 6x/32x
                          /dev/rdsk/c0t6d0   /dev/rdsk/c0t6d0
   disk    1  0/0/1/1.2.0 sdisk  CLAIMED     DEVICE      SEAGATE ST39103LC
                          /dev/rdsk/c1t2d0   /dev/rdsk/c1t2d0
   

   This command displays information about each disk attached to the system, including character raw device names (/dev/rdsk/).

   Note:

   On HP-UX 11i v.3, you can also use agile view to review mass storage devices, including character raw devices (/dev/rdisk/diskxyz). For example:

   #>ioscan -funN -C disk
   Class     I  H/W Path  Driver S/W State   H/W Type     Description
   ===================================================================
   disk      4  64000/0xfa00/0x1   esdisk   CLAIMED     DEVICE       HP 73.4GST373454LC
                    /dev/disk/disk4   /dev/rdisk/disk4
   disk    907  64000/0xfa00/0x2f  esdisk   CLAIMED     DEVICE       COMPAQ  MSA1000 VOLUME
                    /dev/disk/disk907   /dev/rdisk/disk907
   

3. If the ioscan command does not display device name information for a device, enter the following command to install the special device files for any new devices:

   # /usr/sbin/insf -e
   

4. For each disk to add to a disk group, enter the following command on any node to verify that it is not already part of an LVM volume group:

   # /sbin/pvdisplay /dev/dsk/cxtydz
   

   If this command displays volume group information, the disk is already part of a volume group. The disks that you choose must not be part of an LVM volume group.

   Note:

   If you are using different volume management software, for example VERITAS Volume Manager, refer to the appropriate documentation for information about verifying that a disk is not in use.

5. Enter commands similar to the following on every node to change the owner, group, and permissions on the character raw device file for each disk to add to a disk group, so that the owner is the Oracle Grid Infrastructure owner (in this example, grid), and the group that has access is the OSASM group (in this example, asmadmin):

   # chown grid:asmadmin /dev/rdsk/cxtydz
   # chmod 660 /dev/rdsk/cxtydz
   

   Note:

   If you are using a multi-pathing disk driver with ASM, make sure that you set the permissions only on the correct logical device name for the disk.

   If the nodes are configured differently, the device name for a particular device might be different on some nodes. Make sure that you specify the correct device names on each node.

3.3.2.1 Identifying and Using Existing Oracle Database Disk Groups on ASM

The following section describes how to identify existing disk groups and determine the free disk space that they contain.

• Optionally, identify failure groups for the Oracle ASM disk group devices.

  If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

  To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

  Note:

  If you define custom failure groups, then you must specify a minimum of two failure groups for normal redundancy and three failure groups for high redundancy.

3.3.2.2 Creating Disk Groups for Oracle Database Data Files

If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

• All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

• Do not specify multiple partitions on a single physical disk as a disk group device. Oracle ASM expects each disk group device to be on a separate physical disk.

• Although you can specify logical volumes as devices in an Oracle ASM disk group, Oracle does not recommend their use. Non-shared logical volumes are not supported with Oracle RAC. If you want to use logical volumes for your Oracle RAC database, then you must use shared logical volumes created by a cluster-aware logical volume manager.