3 Configuring Storage for Grid Infrastructure for a Cluster and Oracle Real Application Clusters (Oracle RAC)

3.1.1 Overview of Oracle Clusterware and Oracle RAC Storage Options

There are two ways of storing Oracle Clusterware files:

• Oracle Automatic Storage Management (Oracle ASM): You can install Oracle Clusterware files (OCR and voting disks) in Oracle ASM diskgroups.

  Oracle ASM is the required database storage option for Typical installations, and for Standard Edition Oracle RAC installations. It is an integrated, high-performance database file system and disk manager for Oracle Clusterware and Oracle Database files. It performs striping and mirroring of database files automatically.

  Only one Oracle ASM instance is permitted for each node regardless of the number of database instances on the node.

• A supported shared file system: Supported file systems include the following:

  • General Parallel File System (GPFS): A cluster file system for AIX that provides concurrent file access

  • A supported cluster file system. Note that if you intend to use a cluster file system for your data files, then you should create partitions large enough for the database files when you create partitions for Oracle Clusterware.

    See Also:

    The Certification page on My Oracle Support for supported cluster file systems

  • Network File System (NFS): Note that if you intend to use NFS for your data files, then you should create partitions large enough for the database files when you create partitions for Oracle Grid Infrastructure. NFS mounts differ for software binaries, Oracle Clusterware files, and database files.

    Note:

    You can no longer use OUI to install Oracle Clusterware or Oracle Database files on raw disks.

    See Also:

    My Oracle Support for supported file systems and NFS or NAS filers

3.1.2 General Information About Oracle ACFS

Oracle Automatic Storage Management Cluster File System (Oracle ACFS) provides a general purpose file system. You can place Oracle Database binaries on this system, but you cannot place Oracle data files or Oracle Clusterware files on Oracle ACFS. Note the following about Oracle ACFS:

• Oracle Restart does not support root-based Oracle Clusterware resources. For this reason, the following restrictions apply if you run Oracle ACFS on an Oracle Restart Configuration

  • You must manually load and unload Oracle ACFS drivers.

  • You must manually mount and unmount Oracle ACFS file systems, after the Oracle ASM instance is running

  • You can place Oracle ACFS database home file systems into the Oracle ACFS mount registry, along with other registered Oracle ACFS file systems.

• You cannot put Oracle Clusterware binaries and files on Oracle ACFS.

• You cannot put Oracle Database files on Oracle ACFS.

• Oracle ACFS provides a general purpose file system for other files.

3.1.3 General Storage Considerations for Oracle Grid Infrastructure and Oracle RAC

For all installations, you must choose the storage option to use for Oracle Grid Infrastructure (Oracle Clusterware and Oracle ASM), and Oracle Real Application Clusters databases (Oracle RAC). To enable automated backups during the installation, you must also choose the storage option to use for recovery files (the Fast Recovery Area). You do not have to use the same storage option for each file type.

3.1.4 Guidelines for Using Oracle ASM Disk Groups for Storage

During Oracle Grid Infrastructure installation, you can create one disk group. After the Oracle Grid Infrastructure installation, you can create additional disk groups using ASMCA, SQL*Plus, or ASMCMD. Note that with Oracle Database 11g release 2 (11.2) and later releases, Oracle Database Configuration Assistant (DBCA) does not have the functionality to create disk groups for Oracle ASM.

If you install Oracle Database or Oracle RAC after you install Oracle Grid Infrastructure, then you can either use the same disk group for database files, OCR, and voting disk files, or you can use different disk groups. If you create multiple disk groups before installing Oracle RAC or before creating a database, then you can decide to do one of the following:

• Place the data files in the same disk group as the Oracle Clusterware files.

• Use the same Oracle ASM disk group for data files and recovery files.

• Use different disk groups for each file type.

If you create only one disk group for storage, then the OCR and voting disk files, database files, and recovery files are contained in the one disk group. If you create multiple disk groups for storage, then you can choose to place files in different disk groups.

3.1.5 Using Logical Volume Managers with Oracle Grid Infrastructure and Oracle RAC

Oracle Grid Infrastructure and Oracle RAC only support cluster-aware volume managers. This means, the volume manager that you want to use comes with a certain vendor cluster solution. To confirm that a volume manager you want to use is supported, look under the Certifications tab on My Oracle Support whether the associated cluster solution is certified for Oracle RAC. My Oracle Support is available at the following URL:

https://support.oracle.com

3.1.6 Supported Storage Options

The following table shows the storage options supported for storing Oracle Clusterware and Oracle RAC files.

Use the following guidelines when choosing storage options:

• You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.

• You can use Oracle ASM 11g release 2 (11.2) and later to store Oracle Clusterware files. You cannot use prior Oracle ASM releases to do this.

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk locations and at least three Oracle Cluster Registry locations to provide redundancy.

3.1.7 After You Have Selected Disk Storage Options

When you have determined your disk storage options, configure shared storage:

• To use a file system, refer to Section 3.2, "Shared File System Storage Configuration."

• To use Oracle Automatic Storage Management, refer to Section 3.3, "Oracle Automatic Storage Management Storage Configuration."

3.2.1 Requirements for Using a Shared File System

To use a shared file system for Oracle Clusterware, Oracle ASM, and Oracle RAC, the file system must comply with the following requirements:

• To use a cluster file system, it must be a supported cluster file system. Refer to My Oracle Support (https://support.oracle.com) for a list of supported cluster file systems.

• To use an NFS file system, it must be on a supported NAS device. Log in to My Oracle Support, and click the Certification tab to find the most current information about supported NAS devices:

  https://support.oracle.com/

• If you choose to place your Oracle Cluster Registry (OCR) files on a shared file system, then Oracle recommends that one of the following is true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device).

  • At least two file systems are mounted on separate disks, and use the features of Oracle Clusterware 11g release 2 (11.2) to provide redundancy for the OCR.

• If you choose to place your database files on a shared file system, then one of the following should be true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device).

  • The file systems consist of at least two independent file systems on separate physical disks, with the database files on one file system, and the recovery files on a different file system.

• The user account with which you perform the installation (oracle or grid) must have write permissions to create the files in the path that you specify.

Use Table 3-2 and Table 3-3 to determine the minimum size for shared file systems:

In Table 3-2 and Table 3-3, the total required volume size is cumulative. For example, to store all Oracle Clusterware files on the shared file system with normal redundancy, you should have at least 2 GB of storage available over a minimum of three volumes (three separate volume locations for the OCR and two OCR mirrors, and one voting disk on each volume). You should have a minimum of three physical disks, each at least 500 MB, to ensure that voting disks and OCR files are on separate physical disks. If you add Oracle RAC using one volume for database files and one volume for recovery files, then you should have at least 3.5 GB available storage over two volumes, and at least 5.5 GB available total for all volumes.

3.2.2 Deciding to Use a Cluster File System for Oracle Clusterware Files

For new installations, Oracle recommends that you use Oracle Automatic Storage Management (Oracle ASM) to store voting disk and OCR files.

3.2.3 Deciding to Use Direct NFS for Data Files

Direct NFS is an alternative to using kernel-managed NFS. This section contains the following information about Direct NFS:

• About Direct NFS Storage

• Using the Oranfstab File with Direct NFS

• Mounting NFS Storage Devices with Direct NFS

• Specifying Network Paths with the Oranfstab File

3.2.4 Deciding to Use NFS for Data Files

Network-attached storage (NAS) systems use NFS to access data. You can store data files on a supported NFS system.

NFS file systems must be mounted and available over NFS mounts before you start installation. Refer to your vendor documentation to complete NFS configuration and mounting.

Be aware that the performance of Oracle software and databases stored on NAS devices depends on the performance of the network connection between the Oracle server and the NAS device.

For this reason, Oracle recommends that you connect the server to the NAS device using a private dedicated network connection, which should be Gigabit Ethernet or better.

3.2.5 Configuring Storage NFS Mount and Buffer Size Parameters

If you are using NFS for the Grid home or Oracle RAC home, then you must set up the NFS mounts on the storage so that they allow root on the clients mounting to the storage to be considered root instead of being mapped to an anonymous user, and allow root on the client server to create files on the NFS filesystem that are owned by root.

If you are using NFS, then you must set the values for the NFS buffer size parameters rsize and wsize to 32768. The NFS mount options for binaries are:

rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3,timeo=600 

The NFS client-side mount options for Oracle Clusterware files (OCR and voting disk files) are:

cio,rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 

The NFS client-side mount options for Oracle Database datafiles are:

cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600 

Update the /etc/filesystems file on each node with entries similar to the following:

/NFS_mount:
dev = "/vol/gridhome"
vfs = nfs
nodename = /vol/gridhome
mount = true
options = rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3,timeo=600
account = false
/NFS_mount:
dev = "/vol/CWfiles"
vfs = nfs
nodename = /vol/CWfiles
mount = true
options = cio,rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600
account = false
/NFS_mount:
dev = "/vol/datafiles"
vfs = nfs
nodename = /u02/app/oracle/data
mount = true
options = cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600
account = false

Note that mount point options are different for Oracle software binaries, Oracle Clusterware files (OCR and voting disks), and data files.

If you want to create a mount point for binaries only, then enter the following line for a binaries mount point:

nfs_server:/vol/grid /u01/oracle/grid nfs -yes
rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3,timeo=600 

On NFS, you can obtain root access for clients writing to the storage by enabling no_root_squash on the server side. For example, to set up Oracle Clusterware file storage in the path /vol/grid, with nodes node1, node 2, and node3 in the domain mycluster.example.com, add a line similar to the following to the /etc/exports file:

/vol/grid/ node1.mycluster.example.com(rw,no_root_squash)
node2.mycluster.example.com(rw,no_root_squash) node3.mycluster.example.com
(rw,no_root_squash) 

If the domain or DNS is secure so that no unauthorized system can obtain an IP address on it, then you can grant root access by domain, rather than specifying particular cluster member nodes:

For example:

/vol/grid/ *.mycluster.example.com(rw,no_root_squash)

Oracle recommends that you use a secure DNS or domain, and grant root access to cluster member nodes using the domain, as using this syntax allows you to add or remove nodes without the need to reconfigure the NFS server.

If you use Grid Naming Service (GNS), then the subdomain allocated for resolution by GNS within the cluster is a secure domain. Any server without a correctly signed Grid Plug and Play (GPnP) profile cannot join the cluster, so an unauthorized system cannot obtain or use names inside the GNS subdomain.

After changing /etc/exports, reload the file system mount using the following command:

# /usr/sbin/exportfs -avr

3.2.6 Enabling Direct NFS Client Oracle Disk Manager Control of NFS

Complete the following procedure to enable Direct NFS:

1. Create an oranfstab file with the following attributes for each NFS server to be accessed using Direct NFS:

   • Server: The NFS server name.

   • Local: Up to four paths on the database host, specified by IP address or by name, as displayed using the ifconfig command run on the database host.

   • Path: Up to four network paths to the NFS server, specified either by IP address, or by name, as displayed using the ifconfig command on the NFS server.

   • Export: The exported path from the NFS server.

   • Mount: The corresponding local mount point for the exported volume.

   • Mnt_timeout: Specifies (in seconds) the time Direct NFS client should wait for a successful mount before timing out. This parameter is optional. The default timeout is 10 minutes (600).

   • Dontroute: Specifies that outgoing messages should not be routed by the operating system, but instead sent using the IP address to which they are bound.

   The examples that follow show three possible NFS server entries in oranfstab. A single oranfstab can have multiple NFS server entries.

   Example 3-1 Using Local and Path NFS Server Entries

   The following example uses both local and path. Since they are in different subnets, we do not have to specify dontroute.

   server: MyDataServer1
   local: 192.0.2.0
   path: 192.0.2.1
   local: 192.0.100.0
   path: 192.0.100.1
   export: /vol/oradata1 mount: /mnt/oradata1
   

   Example 3-2 Using Local and Path in the Same Subnet, with dontroute

   The following example shows local and path in the same subnet. dontroute is specified in this case:

   server: MyDataServer2
   local: 192.0.2.0
   path: 192.0.2.128
   local: 192.0.2.1
   path: 192.0.2.129
   dontroute
   export: /vol/oradata2 mount: /mnt/oradata2
   

   Example 3-3 Using Names in Place of IP Addresses, with Multiple Exports

   server: MyDataServer3
   local: LocalPath1
   path: NfsPath1
   local: LocalPath2
   path: NfsPath2
   local: LocalPath3
   path: NfsPath3
   local: LocalPath4
   path: NfsPath4
   dontroute
   export: /vol/oradata3 mount: /mnt/oradata3
   export: /vol/oradata4 mount: /mnt/oradata4
   export: /vol/oradata5 mount: /mnt/oradata5
   export: /vol/oradata6 mount: /mnt/oradata6
   

2. By default, Direct NFS is installed in a disabled state. To enable Direct NFS, complete the following steps on each node. If you use a shared Grid home for the cluster, then complete the following steps in the shared Grid home:

   1. Log in as the Oracle Grid Infrastructure installation owner.

   2. Change directory to Grid_home/rdbms/lib.

   3. Enter the following commands:

      $ make -f ins_rdbms.mk dnfs_on
      

3.2.7 Configuring HACMP Multinode Disk Heartbeat (MNDHB)

This section contains the following topics:

• Overview of Requirements for Using HACMP with Oracle Clusterware

• Deploying HACMP and MDNDHB for Oracle Clusterware

• Upgrading an Existing Oracle Clusterware and HACMP Installation

3.2.8 Configuring Raw Logical Volumes for Oracle Clusterware

This section describes how to configure raw logical volumes for Oracle Clusterware and database file storage. The procedures in this section describe how to create a new volume group that contains the logical volumes required for both types of files.

Before you continue, review the following guidelines which contain important information about using volume groups with this release of Oracle RAC:

• You must use concurrent-capable volume groups for Oracle Clusterware.

• The Oracle Clusterware files require less than 560 MB of disk space, with external redundancy. To make efficient use of the disk space in a volume group, Oracle recommends that you use the same volume group for the logical volumes for both the Oracle Clusterware files and the database files.

• If you are upgrading an existing Oracle9i release 2 Oracle RAC installation that uses raw logical volumes, then you can use the existing SRVM configuration repository logical volume for the OCR and create a new logical volume in the same volume group for the Oracle Clusterware voting disk. However, you must remove this volume group from the HACMP concurrent resource group that activates it before you install Oracle Clusterware.

  See Also:

  The HACMP documentation for information about removing a volume group from a concurrent resource group.

  Note:

  If you are upgrading a database, then you must also create a new logical volume for the SYSAUX tablespace. Refer to the "Configuring New Oracle Clusterware Volume Group Raw Logical Volumes" section for more information about the requirements for the Oracle Clusterware voting disk and SYSAUX logical volumes.

• You must use a HACMP concurrent resource group to activate new or existing volume groups that contain only database files (not Oracle Clusterware files).

  See Also:

  The HACMP documentation for information about adding a volume group to a new or existing concurrent resource group.

• All volume groups that you intend to use for Oracle Clusterware must be activated in concurrent mode before you start the installation.

• The procedures in this section describe how to create basic volumes groups and volumes. If you want to configure more complex volumes, (using mirroring, for example), then use this section in conjunction with the HACMP documentation.

3.2.9 Configuring New Oracle Clusterware Volume Group Raw Logical Volumes

To create the required raw logical volumes in the new Oracle Clusterware volume group:

1. Identify the logical volumes that you must create.

2. If you prefer, you can also use the command smit mklv to create raw logical volumes.

   The following example shows the command used to create a logical volume for the ocr volume group in the SYSAUX tablespace with a physical partition size of 114 MB (1792/7 = 256):

   # /usr/sbin/mklv -y test_sysaux_raw_1792m -T O -w n -s n -r n ocr 7
   

3. Change the owner, group, and permissions on the character device files associated with the logical volumes that you created, as follows:

   Note:

   The device file associated with the Oracle Cluster Registry must be owned by root. All other device files must be owned by the Oracle software owner user (oracle).

   # chown oracle:dba /dev/rora_vote_raw_280m
   # chmod 660 /dev/rora_vote_raw_280m
   # chown root:oinstall /dev/rora_ocr_raw_280m
   # chmod 640 /dev/rora_ocr_raw_280m
   

3.2.10 Creating a Volume Group for Database Files

To create a volume group for the Oracle Database files:

1. If necessary, install the shared disks that you intend to use.

2. To ensure that the disks are available, enter the following command on every node:

   # /usr/sbin/lsdev -Cc disk
   

   The output from this command is similar to the following:

   hdisk0 Available 1A-09-00-8,0  16 Bit LVD SCSI Disk Drive
   hdisk1 Available 1A-09-00-9,0  16 Bit LVD SCSI Disk Drive
   hdisk2 Available 17-08-L       SSA Logical Disk Drive
   

3. If a disk is not listed as available on any node, then enter the following command to configure the new disks:

   # /usr/sbin/cfgmgr
   

4. Enter the following command on any node to identify the device names and any associated volume group for each disk:

   # /usr/sbin/lspv
   

   The output from this command is similar to the following:

   hdisk0     0000078752249812   rootvg
   hdisk1     none               none
   hdisk4     00034b6fd4ac1d71   ccvg1
   

   For each disk, this command shows:

   • The disk device name

   • Either the 16 character physical volume identifier (PVID) if the disk has one, or none

   • Either the volume group to which the disk belongs, or none

   The disks that you want to use may have a PVID, but they must not belong to existing volume groups.

5. If a disk that you want to use for the volume group does not have a PVID, then enter a command similar to the following to assign one to it:

   # /usr/sbin/chdev -l hdiskn -a pv=yes
   

6. To identify used device major numbers, enter the following command on each node of the cluster:

   # ls -la /dev | more
   

   This command displays information about all configured devices, similar to the following:

   crw-rw----   1 root     system    45,  0 Jul 19 11:56 vg1
   

   In this example, 45 is the major number of the vg1 volume group device.

7. Identify an appropriate major number that is unused on all nodes in the cluster.

8. To create a volume group, enter a command similar to the following, or use SMIT (smit mkvg):

   # /usr/sbin/mkvg -y VGname -B -s PPsize -V majornum -n \
   -C PhysicalVolumes
   

9. The following table describes the options and variables used in this example. Refer to the mkvg man page for more information about these options.

   Command Option	SMIT Field	Sample Value and Description
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or it could specify the name of the database that you intend to create.
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or for a database volume group, it could specify the name of the database that you intend to create.
   -B	Create a big VG format Volume Group	Specify this option to create a big VG format volume group. Note: If you are using SMIT, then choose yes for this field.
   -s PPsize	Physical partition SIZE in megabytes	32 Specify the size of the physical partitions for the database. The sample value shown enables you to include a disk up to 32 GB in size (32 MB * 1016).
   -V Majornum	Volume Group MAJOR NUMBER	46 Specify the device major number for the volume group that you identified in Step 7.
   -n	Activate volume group AUTOMATICALLY at system restart	Specify this option to prevent the volume group from being activated at system restart. Note: If you are using SMIT, then choose no for this field.
   -C	Create VG Concurrent Capable	Specify this option to create a concurrent capable volume group. Note: If you are using SMIT, then choose yes for this field.
   PhysicalVolumes	PHYSICAL VOLUME names	hdisk3 hdisk4 Specify the device names of the disks that you want to add to the volume group.

   
   

10. Enter a command similar to the following to vary on the volume group that you created:

    # /usr/sbin/varyonvg VGname
    

3.2.11 Creating a Volume Group for Oracle Clusterware

To create a volume group for the Oracle Clusterware files:

1. If necessary, install the shared disks that you intend to use.

2. To ensure that the disks are available, enter the following command on every node:

   # /usr/sbin/lsdev -Cc disk
   

   The output from this command is similar to the following:

   hdisk0 Available 1A-09-00-8,0  16 Bit LVD SCSI Disk Drive
   hdisk1 Available 1A-09-00-9,0  16 Bit LVD SCSI Disk Drive
   hdisk2 Available 17-08-L       SSA Logical Disk Drive
   

3. If a disk is not listed as available on any node, then enter the following command to configure the new disks:

   # /usr/sbin/cfgmgr
   

4. Enter the following command on any node to identify the device names and any associated volume group for each disk:

   # /usr/sbin/lspv
   

   The output from this command is similar to the following:

   hdisk0     0000078752249812   rootvg
   hdisk1     none               none
   hdisk4     00034b6fd4ac1d71   ccvg1
   

   For each disk, this command shows:

   • The disk device name

   • Either the 16 character physical volume identifier (PVID) if the disk has one, or none

   • Either the volume group to which the disk belongs, or none

   The disks that you want to use may have a PVID, but they must not belong to existing volume groups.

5. If a disk that you want to use for the volume group does not have a PVID, then enter a command similar to the following to assign one to it:

   # /usr/sbin/chdev -l hdiskn -a pv=yes
   

6. To identify used device major numbers, enter the following command on each node of the cluster:

   # ls -la /dev | more
   

   This command displays information about all configured devices, similar to the following:

   crw-rw----   1 root     system    45,  0 Jul 19 11:56 vg1
   

   In this example, 45 is the major number of the vg1 volume group device.

7. Identify an appropriate major number that is unused on all nodes in the cluster.

8. To create a volume group, enter a command similar to the following, or use SMIT (smit mkvg):

   # /usr/sbin/mkvg -y VGname -B -s PPsize -V majornum -n \
   -C PhysicalVolumes
   

9. The following table describes the options and variables used in this example. Refer to the mkvg man page for more information about these options.

   Command Option	SMIT Field	Sample Value and Description
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or it could specify the name of the database that you intend to create.
   -y VGname	VOLUME GROUP name	oracle_vg1 Specify the name for the volume group. The name that you specify could be a generic name, as shown, or for a database volume group, it could specify the name of the database that you intend to create.
   -B	Create a big VG format Volume Group	Specify this option to create a big VG format volume group. Note: If you are using SMIT, then choose yes for this field.
   -s PPsize	Physical partition SIZE in megabytes	32 Specify the size of the physical partitions for the database. The sample value shown enables you to include a disk up to 32 GB in size (32 MB * 1016).
   -V Majornum	Volume Group MAJOR NUMBER	46 Specify the device major number for the volume group that you identified in Step 7.
   -n	Activate volume group AUTOMATICALLY at system restart	Specify this option to prevent the volume group from being activated at system restart. Note: If you are using SMIT, then choose no for this field.
   -C	Create VG Concurrent Capable	Specify this option to create a concurrent capable volume group. Note: If you are using SMIT, then choose yes for this field.
   PhysicalVolumes	PHYSICAL VOLUME names	hdisk3 hdisk4 Specify the device names of the disks that you want to add to the volume group.

   
   

10. Enter a command similar to the following to vary on the volume group that you created:

    # /usr/sbin/varyonvg VGname
    

3.2.12 Importing the Volume Group on the Other Cluster Nodes

To make the volume group available to all nodes in the cluster, you must import it on each node, as follows:

1. Because the physical volume names may be different on the other nodes, enter the following command to determine the PVID of the physical volumes used by the volume group:

   # /usr/sbin/lspv
   

2. Note the PVIDs of the physical devices used by the volume group.

3. To vary off the volume group that you want to use, enter a command similar to the following on the node where you created it:

   # /usr/sbin/varyoffvg VGname
   

4. On each cluster node, complete the following steps:

   1. Enter the following command to determine the physical volume names associated with the PVIDs you noted previously:

      # /usr/sbin/lspv
      

   2. On each node of the cluster, enter commands similar to the following to import the volume group definitions:

      # /usr/sbin/importvg -y VGname -V MajorNumber PhysicalVolume
      

      In this example, MajorNumber is the device major number for the volume group and PhysicalVolume is the name of one of the physical volumes in the volume group.

      For example, to import the definition of the oracle_vg1 volume group with device major number 45 on the hdisk3 and hdisk4 physical volumes, enter the following command:

      # /usr/sbin/importvg -y oracle_vg1 -V 45 hdisk3
      

   3. Change the owner, group, and permissions on the character device files associated with the logical volumes you created, as follows:

      # chown oracle:dba /dev/rora_vote_raw_280m
      # chmod 660 /dev/rora_vote_raw_280m
      # chown root:oinstall /dev/rora_ocr_raw_280m
      # chmod 640 /dev/rora_ocr_raw_280m
      

   4. Enter the following command to ensure that the volume group will not be activated by the operating system when the node starts:

      # /usr/sbin/chvg -a n VGname
      

3.2.13 Activating the Volume Group in Concurrent Mode on All Cluster Nodes

To activate the volume group in concurrent mode on all cluster nodes, enter the following command on each node:

# /usr/sbin/varyonvg -c VGname

3.2.14 Creating Directories for Oracle Clusterware Files on Shared File Systems

Use the following instructions to create directories for Oracle Clusterware files. You can also configure shared file systems for the Oracle Database and recovery files.

To create directories for the Oracle Clusterware files on separate file systems from the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   Note:

   The mount point that you use for the file system must be identical on each node. Make sure that the file systems are configured to mount automatically when a node restarts.

2. Use the df -k command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems that you want to use:

   File Type	File System Requirements
   Oracle Clusterware files	Choose a file system with at least 600 MB of free disk space (one OCR and one voting disk, with external redundancy).
   Database files	Choose either: A single file system with at least 1.5 GB of free disk space. Two or more file systems with at least 1.5 GB of free disk space in total.
   Recovery files	Choose a file system with at least 2 GB of free disk space.

   
   

   If you are using the same file system for more than one type of file, then add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. If the user performing installation has permissions to create directories on the storage location where you plan to install Oracle Clusterware, then OUI creates the Oracle Clusterware file directory.

   1. If necessary, configure the shared file systems to use and mount them on each node.

      Note:

      The mount point that you use for the file system must be identical on each node. Ensure that the file systems are configured to mount automatically when a node restarts.

   2. Use the df command to determine the free disk space on each mounted file system.

   3. From the display, identify the file systems to use. Choose a file system with a minimum of 600 MB of free disk space (one OCR and one voting disk, with external redundancy).

      If you are using the same file system for multiple file types, then add the disk space requirements for each type to determine the total disk space requirement.

      Note:

      If you are using GPFS, then you can use the same file system for all purposes. This includes using it for the Oracle home directory and for storing data files and logs. For optimal performance, you should use a large GPFS block size (typically, at least 512 KB). GPFS is designed for scalability, and there is no requirement to create multiple GPFS file systems as long as the amount of data fits in a single GPFS file system.

   4. Note the names of the mount point directories for the file systems that you identified.

   5. If the user performing installation (typically, grid or oracle) has permissions to create directories on the storage location where you plan to install Oracle Clusterware files, then OUI creates the Oracle Clusterware file directory.

      If the user performing installation does not have write access, then you must create these directories manually using commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on the directory. For example, where the user is oracle, and the Oracle Clusterware file storage area is cluster:

      # mkdir /mount_point/cluster
      # chown oracle:oinstall /mount_point/cluster
      # chmod 775 /mount_point/cluster
      

      Note:

      After installation, directories in the installation path for the Oracle Cluster Registry (OCR) files should be owned by root, and not writable by any account other than root.

When you have completed creating subdirectories in each of the mount point directories, and set the appropriate owner, group, and permissions, you have completed GPFS configuration.

3.2.15 Creating Directories for Oracle Database Files on Shared File Systems

Use the following instructions to create directories for shared file systems for Oracle Database and recovery files (for example, for an Oracle RAC database).

1. If necessary, configure the shared file systems and mount them on each node.

   Note:

   The mount point that you use for the file system must be identical on each node. Ensure that the file systems are configured to mount automatically when a node restarts.

2. Use the df -k command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems:

   File Type	File System Requirements
   Database files	Choose either: A single file system with at least 1.5 GB of free disk space. Two or more file systems with at least 1.5 GB of free disk space in total.
   Recovery files	Choose a file system with at least 2 GB of free disk space.

   
   

   If you are using the same file system for multiple file types, then add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. If the user performing installation (typically, oracle) has permissions to create directories on the disks where you plan to install Oracle Database, then DBCA creates the Oracle Database file directory, and the Recovery file directory.

   If the user performing installation does not have write access, then you must create these directories manually using commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on them:

   • Database file directory:

     # mkdir /mount_point/oradata
     # chown oracle:oinstall /mount_point/oradata
     # chmod 775 /mount_point/oradata
     

   • Recovery file directory (Fast Recovery Area):

     # mkdir /mount_point/fast_recovery_area
     # chown oracle:oinstall /mount_point/fast_recovery_area
     # chmod 775 /mount_point/fast_recovery_area
     

By making members of the oinstall group owners of these directories, this permits them to be read by multiple Oracle homes, including those with different OSDBA groups.

When you have completed creating subdirectories in each of the mount point directories, and set the appropriate owner, group, and permissions, you have completed NFS configuration for Oracle Database shared storage.

3.2.16 Disabling Direct NFS Client Oracle Disk Management Control of NFS

Complete the following steps to disable the Direct NFS client:

1. Log in as the Oracle Grid Infrastructure installation owner, and disable the Direct NFS client using the following commands, where Grid_home is the path to the Oracle Grid Infrastructure home:

   $ cd Grid_home/rdbms/lib
   $ make -f ins_rdbms.mk dnfs_off
   

   Enter these commands on each node in the cluster, or on the shared Grid home if you are using a shared home for the Oracle Grid Infrastructure installation.

2. Remove the oranfstab file.

3.3.1 Configuring Storage for Oracle Automatic Storage Management

This section describes how to configure storage for use with Oracle Automatic Storage Management (Oracle ASM).

• Identifying Storage Requirements for Oracle ASM

• Creating Files on a NAS Device for Use with Oracle ASM

3.3.2 Using an Existing Oracle ASM Disk Group

Select from the following choices to store either database or recovery files in an existing Oracle ASM disk group, depending on installation method:

• If you select an installation method that runs Database Configuration Assistant in interactive mode, then you can decide whether you want to create a disk group, or to use an existing one.

  The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

• If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine if an existing Oracle ASM disk group exists, or to determine if there is sufficient disk space in a disk group, you can use the Oracle ASM command line tool (asmcmd), Oracle Enterprise Manager Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine if an Oracle ASM instance is configured on the system:

   $ more /etc/oratab
   

   If an Oracle ASM instance is configured on the system, then the oratab file should contain a line similar to the following:

   +ASM2:oracle_home_path
   

   In this example, +ASM2 is the system identifier (SID) of the Oracle ASM instance, with the node number appended, and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an Oracle ASM instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the Oracle ASM instance.

3. Connect to the Oracle ASM instance and start the instance if necessary:

   $ $ORACLE_HOME/bin/asmcmd
   ASMCMD> startup
   

4. Enter one of the following commands to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   ASMCMD> lsdb
   

   or:

   $ORACLE_HOME/bin/asmcmd -p lsdg
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   Note:

   If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

3.3.3 Configuring Disk Devices for Oracle ASM

You can configure raw disks for use as Oracle Automatic Storage Management (Oracle ASM) disk groups. To use Oracle ASM with raw disks, you must create sufficient partitions for your data files, and then bind the partitions to raw disks. Make a list of the raw disk names you create for the data files, and have the list available during database installation.

In the following procedure, you are directed to set physical volume IDs (PVIDs) for raw disks. Oracle recommends that you complete the entire procedure, even if you are certain that you do not have PVIDs configured on your system, to prevent the possibility of configuration issues.

/dev/rdlmfdrv8
/dev/rdlmfdrv9

Use the following procedure to configure disks:

1. If necessary, install the disks that you intend to use for the disk group and restart the system.

2. Identify or create the disks that you want to include in the Oracle ASM disk group. As the root user, enter the following command on any node to identify the device names for the disk devices that you want to use:

   # /usr/sbin/lspv | grep -i none 
   

   This command displays information similar to the following for each disk device that is not configured in a volume group:

   hdisk17         0009005fb9c23648                    None  
   

   In this example, hdisk17 is the device name of the disk and 0009005fb9c23648 is the physical volume ID (PVID).

3. If a disk device that you want to use does not have a PVID, then enter a command similar to the following to assign one to it, where n is the number of the hdisk:

   # chdev -l hdiskn -a pv=yes
   

   Note:

   If you have an existing PVID, then chdev overwrites the existing PVID. Be aware that if you have applications depending on the previous PVID, then they will fail.

4. On each of the other nodes, enter a command similar to the following to identify the device name associated with each PVID on that node:

   # /usr/sbin/lspv | grep -i "0009005fb9c23648"
   

   The output from this command should be similar to the following:

   hdisk18         0009005fb9c23648                    None
   

   In this example, the device name associated with the disk device (hdisk18) is different on this node.

5. If the device names are the same on all nodes, then enter commands similar to the following on all nodes to change the owner, group, and permissions on the character raw device files for the disk devices where grid is the Oracle Grid Infrastructure installation owner, and asmadmin is the OSASM group:

   # chown grid:asmadmin /dev/rhdiskn
   # chmod 660 /dev/rhdiskn
   

6. To enable simultaneous access to a disk device from multiple nodes, you must set the appropriate Object Data Manager (ODM) attribute, depending on the type of reserve attribute used by your disks. The following section describes how to perform this task using hdisk logical names. Refer to your operating system documentation to find logical device names.

   To determine the reserve setting your disks use, enter the following command, where n is the hdisk device number:

   # lsattr -E -l hdiskn | grep reserve_
   

   The response is either a reserve_lock setting, or a reserve_policy setting. If the attribute is reserve_lock, then ensure that the setting is reserve_lock = no. If the attribute is reserve_policy, then ensure that the setting is reserve_policy = no_reserve.

   If necessary, change the setting with the chdev command using the following syntax, where n is the hdisk device number:

   chdev -l hdiskn -a [ reserve_lock=no | reserve_policy=no_reserve ]
   

   For example, to change a setting for the device hdisk4 from reserve_lock=yes to reserve_lock=no, enter the following command:

   # chdev -l hdisk4  -a  reserve_lock=no
   

   To verify that the setting is correct on all disk devices, enter the following command:

   # lsattr -El hdiskn | grep reserve
   

7. Enter commands similar to the following on any node to clear the PVID from each disk device that you want to use:

   # /usr/sbin/chdev -l hdiskn -a pv=clear
   

   When you are installing Oracle Clusterware, you must enter the paths to the appropriate device files when prompted for the path of the OCR and Oracle Clusterware voting disk. For example:

   /dev/rhdisk10
   

3.3.4 Using Disk Groups with Oracle Database Files on Oracle ASM

Review the following sections to configure Oracle Automatic Storage Management storage for Oracle Clusterware and Oracle Database Files:

• Using Existing Oracle Database Disk Groups on Oracle ASM

• Creating Disk Groups for Oracle Database Data Files

3.3.5 Configuring Oracle Automatic Storage Management Cluster File System

Oracle ACFS is installed as part of an Oracle Grid Infrastructure installation (Oracle Clusterware and Oracle Automatic Storage Management) for 11g release 2 (11.2). You can configure Oracle ACFS for a database home, or use ASMCA to configure ACFS as a general purpose file system.

To configure Oracle ACFS for an Oracle Database home for an Oracle RAC database:

1. Install Oracle Grid Infrastructure for a cluster (Oracle Clusterware and Oracle Automatic Storage Management)

2. Change directory to the Oracle Grid Infrastructure home. For example:

   $ cd /u01/app/11.2.0/grid
   

3. Ensure that the Oracle Grid Infrastructure installation owner has read and write permissions on the storage mountpoint you want to use. For example, if you want to use the mountpoint /u02/acfsmounts/:

   $ ls -l /u02/acfsmounts
   

4. Start Oracle ASM Configuration Assistant as the grid installation owner. For example:

   ./asmca
   

5. The Configure ASM: ASM Disk Groups page shows you the Oracle ASM disk group you created during installation. Click the ASM Cluster File Systems tab.

6. On the ASM Cluster File Systems page, right-click the Data disk, then select Create ACFS for Database Home.

7. In the Create ACFS Hosted Database Home window, enter the following information:

   • Database Home ADVM Volume Device Name: Enter the name of the database home. The name must be unique in your enterprise. For example: dbase_01

   • Database Home Mountpoint: Enter the directory path for the mount point. For example: /u02/acfsmounts/dbase_01

     Make a note of this mount point for future reference.

   • Database Home Size (GB): Enter in gigabytes the size you want the database home to be.

   • Database Home Owner Name: Enter the name of the Oracle Database installation owner you plan to use to install the database. For example: oracle1

   • Database Home Owner Group: Enter the OSDBA group whose members you plan to provide when you install the database. Members of this group are given operating system authentication for the SYSDBA privileges on the database. For example: dba1

   • Click OK when you have completed your entries.

8. Run the script generated by Oracle ASM Configuration Assistant as a privileged user (root). On an Oracle Clusterware environment, the script registers the ACFS as a resource managed by Oracle Clusterware. Registering ACFS as a resource helps Oracle Clusterware to mount the ACFS automatically in proper order when ACFS is used for an Oracle RAC database Home.

9. During Oracle RAC installation, ensure that you or the DBA who installs Oracle RAC selects for the Oracle home the mount point you provided in the Database Home Mountpoint field (in the preceding example, /u02/acfsmounts/dbase_01).

3.3.6 Upgrading Existing Oracle ASM Instances

If you have an Oracle ASM installation from a prior release installed on your server, or in an existing Oracle Clusterware installation, then you can use Oracle Automatic Storage Management Configuration Assistant (ASMCA, located in the path Grid_home/bin) to upgrade the existing Oracle ASM instance to 11g release 2 (11.2), and subsequently configure failure groups, Oracle ASM volumes and Oracle Automatic Storage Management Cluster File System (Oracle ACFS).

During installation, if you are upgrading from an Oracle ASM release prior to 11.2, and you chose to use Oracle ASM and ASMCA detects that there is a prior Oracle ASM version installed in another Oracle ASM home, then after installing the Oracle ASM 11g release 2 (11.2) binaries, you can start ASMCA to upgrade the existing Oracle ASM instance. You can then choose to configure an Oracle ACFS deployment by creating Oracle ASM volumes and using the upgraded Oracle ASM to create the Oracle ACFS.

If you are upgrading from Oracle ASM 11g release 2 (11.2.0.1) or later, then Oracle ASM is always upgraded with Oracle Grid Infrastructure as part of the rolling upgrade, and ASMCA is started by the root scripts during upgrade. ASMCA cannot perform a separate upgrade of Oracle ASM from release 11.2.0.1 to 11.2.0.2.

On an existing Oracle Clusterware or Oracle RAC installation, if the prior version of Oracle ASM instances on all nodes is 11g release 1, then you are provided with the option to perform a rolling upgrade of Oracle ASM instances. If the prior version of Oracle ASM instances on an Oracle RAC installation are from a release prior to 11g release 1, then rolling upgrades cannot be performed. Oracle ASM on all nodes will be upgraded to 11g release 2 (11.2).