3 Configuring Storage for Grid Infrastructure for a Cluster and Oracle Real Application Clusters (Oracle RAC)

3.1.4.1 General Storage Considerations for Oracle Clusterware

Oracle Clusterware voting disks are used to monitor cluster node status, and Oracle Cluster Registry (OCR) files contain configuration information about the cluster. You can place voting disks and OCR files either in an Oracle ASM disk group, or on a cluster file system or shared network file system. Storage must be shared; any node that does not have access to an absolute majority of voting disks (more than half) will be restarted.

3.1.4.2 General Storage Considerations for Oracle RAC

Use the following guidelines when choosing the storage options to use for each file type:

• You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.

• If you plan to install an Oracle RAC home on a shared OCFS2 location, then you must upgrade OCFS2 to at least version 1.4.1, which supports shared writable mmaps.

• Oracle recommends that you choose Oracle ASM as the storage option for database and recovery files.

• For Standard Edition Oracle RAC installations, Oracle ASM is the only supported storage option for database or recovery files.

• If you intend to use Oracle ASM with Oracle RAC, and you are configuring a new Oracle ASM instance, then your system must meet the following conditions:

  • All nodes on the cluster have Oracle Clusterware and Oracle ASM 11g release 2 (11.2) installed as part of an Oracle Grid Infrastructure for a cluster installation.

  • Any existing Oracle ASM instance on any node in the cluster is shut down.

• Raw or block devices are supported only when upgrading an existing installation using the partitions already configured. On new installations, using raw or block device partitions is not supported by Oracle Automatic Storage Management Configuration Assistant (ASMCA) or Oracle Universal Installer (OUI), but is supported by the software if you perform manual configuration.

  See Also:

  Oracle Database Upgrade Guide for information about how to prepare for upgrading an existing database

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy.

3.2.3.1 About Direct NFS Storage

With Oracle Database 11g release 2 (11.2), instead of using the operating system kernel NFS client, you can configure Oracle Database to access NFS V3 servers directly using an Oracle internal Direct NFS client.

To enable Oracle Database to use Direct NFS, the NFS file systems must be mounted and available over regular NFS mounts before you start installation. Direct NFS manages settings after installation. You should still set the kernel mount options as a backup, but for normal operation, Direct NFS will manage NFS mounts.

Refer to your vendor documentation to complete NFS configuration and mounting.

Some NFS file servers require NFS clients to connect using reserved ports. If your filer is running with reserved port checking, then you must disable it for Direct NFS to operate. To disable reserved port checking, consult your NFS file server documentation.

3.2.3.2 Using the Oranfstab File with Direct NFS

If you use Direct NFS, then you can choose to use a new file specific for Oracle data file management, oranfstab, to specify additional options specific for Oracle Database to Direct NFS. For example, you can use oranfstab to specify additional paths for a mount point. You can add the oranfstab file either to /etc or to $ORACLE_HOME/dbs.

With shared Oracle homes, when the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database. In this case, all nodes running an Oracle RAC database use the same $ORACLE_HOME/dbs/oranfstab file. In non-shared Oracle RAC installs, oranfstab must be replicated on all nodes.

When the oranfstab file is placed in /etc, then it is globally available to all Oracle databases, and can contain mount points used by all Oracle databases running on nodes in the cluster, including standalone databases. However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file /etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the /etc/fstab file.

In all cases, mount points must be mounted by the kernel NFS system, even when they are being served using Direct NFS.

3.2.3.3 Mounting NFS Storage Devices with Direct NFS

Direct NFS determines mount point settings to NFS storage devices based on the configurations in /etc/mtab, which are changed with configuring the /etc/fstab file.

Direct NFS searches for mount entries in the following order:

1. $ORACLE_HOME/dbs/oranfstab

2. /etc/oranfstab

3. /etc/mtab

Direct NFS uses the first matching entry found.

Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, complete the following steps:

1. Change the directory to $ORACLE_HOME/rdbms/lib.

2. Enter the following command:

   make -f ins_rdbms.mk dnfs_on
   

If Oracle Database uses Direct NFS mount points configured using oranfstab, then it first verifies kernel NFS mounts by cross-checking entries in oranfstab with operating system NFS mount points. If a mismatch exists, then Direct NFS logs an informational message, and does not operate.

If Oracle Database cannot open an NFS server using Direct NFS, then Oracle Database uses the platform operating system kernel NFS client. In this case, the kernel NFS mount options must be set up as defined in "Checking NFS Mount and Buffer Size Parameters for Oracle RAC". Additionally, an informational message is logged into the Oracle alert and trace files indicating that Direct NFS could not be established.

Section 3.1.7, "Supported Storage Options" lists the file types that are supported by Direct NFS.

The Oracle files resident on the NFS server that are served by the Direct NFS Client are also accessible through the operating system kernel NFS client.

3.2.3.4 Specifying Network Paths with the Oranfstab File

Direct NFS can use up to four network paths defined in the oranfstab file for an NFS server. The Direct NFS client performs load balancing across all specified paths. If a specified path fails, then Direct NFS reissues I/O commands over any remaining paths.

Use the following SQL*Plus views for managing Direct NFS in a cluster environment:

• gv$dnfs_servers: Shows a table of servers accessed using Direct NFS.

• gv$dnfs_files: Shows a table of files currently open using Direct NFS.

• gv$dnfs_channels: Shows a table of open network paths (or channels) to servers for which Direct NFS is providing files.

• gv$dnfs_stats: Shows a table of performance statistics for Direct NFS.

3.3.1.1 Identifying Storage Requirements for Oracle Automatic Storage Management

To identify the storage requirements for using Oracle ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use Oracle ASM for Oracle Clusterware files (OCR and voting disks), Oracle Database files, recovery files, or all files except for Oracle Clusterware or Oracle Database binaries. Oracle Database files include data files, control files, redo log files, the server parameter file, and the password file.

   Note:

   You do not have to use the same storage mechanism for Oracle Clusterware, Oracle Database files and recovery files. You can use a shared file system for one file type and Oracle ASM for the other.

   If you choose to enable automated backups and you do not have a shared file system available, then you must choose Oracle ASM for recovery file storage.

   If you enable automated backups during the installation, then you can select Oracle ASM as the storage mechanism for recovery files by specifying an Oracle Automatic Storage Management disk group for the Fast Recovery Area. If you select a noninteractive installation mode, then by default it creates one disk group and stores the OCR and voting disk files there. If you want to have any other disk groups for use in a subsequent database install, then you can choose interactive mode, or run ASMCA (or a command line tool) to create the appropriate disk groups before starting the database install.

2. Choose the Oracle ASM redundancy level to use for the Oracle ASM disk group.

   The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group and determines the number of disks and amount of free disk space that you require, as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     For Oracle Clusterware files, External redundancy disk groups provide 1 voting disk file, and 1 OCR, with no copies. You must use an external technology to provide mirroring for high availability.

     Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.

   • Normal redundancy

     In a normal redundancy disk group, to increase performance and reliability, Oracle ASM by default uses two-way mirroring. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For Oracle Clusterware files, Normal redundancy disk groups provide 3 voting disk files, 1 OCR and 2 copies (one primary and one secondary mirror). With normal redundancy, the cluster can survive the loss of one failure group.

     For most installations, Oracle recommends that you select normal redundancy.

   • High redundancy

     In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     For Oracle Clusterware files, High redundancy disk groups provide 5 voting disk files, 1 OCR and 3 copies (one primary and two secondary mirrors). With high redundancy, the cluster can survive the loss of two failure groups.

     While high redundancy disk groups do provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to select high redundancy disk groups.

3. Determine the total amount of disk space that you require for Oracle Clusterware files, and for the database files and recovery files.

   Use Table 3-5 and Table 3-6 to determine the minimum number of disks and the minimum disk space requirements for installing Oracle Clusterware files, and installing the starter database, where you have voting disks in a separate disk group:

   Table 3-5 Total Oracle Clusterware Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Oracle Cluster Registry (OCR) Files	Voting Disk Files	Both File Types
   External	1	300 MB	300 MB	600 MB
   Normal	3	600 MB	900 MB	1.5 GBFoot 1
   High	5	900 MB	1.5 GB	2.4 GB

   
   

   ^Footnote 1 If you create a disk group during installation, then it must be at least 2 GB.

   Note:

   If the voting disk files are in a disk group, be aware that disk groups with Oracle Clusterware files (OCR and voting disk files) have a higher minimum number of failure groups than other disk groups.

   If you create a disk group as part of the installation in order to install the OCR and voting disk files, then the installer requires that you create these files on a disk group with at least 2 GB of available space.

   A quorum failure group is a special type of failure group and disks in these failure groups do not contain user data. A quorum failure group is not considered when determining redundancy requirements in respect to storing user data. However, a quorum failure group counts when mounting a disk group.

   Table 3-6 Total Oracle Database Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Database Files	Recovery Files	Both File Types
   External	1	1.5 GB	3 GB	4.5 GB
   Normal	2	3 GB	6 GB	9 GB
   High	3	4.5 GB	9 GB	13.5 GB

   
   

4. For Oracle Clusterware installations, you must also add additional disk space for the Oracle ASM metadata. You can use the following formula to calculate the disk space requirements (in MB) for OCR and voting disk files, and the Oracle ASM metadata:

   total = [2 * ausize * disks] + [redundancy * (ausize * (nodes * (clients + 1) + 30) + (64 * nodes) + 533)]

   Where:

   • redundancy = Number of mirrors: external = 1, normal = 2, high = 3.

   • ausize = Metadata AU size in megabytes (default is 1 MB)

   • nodes = Number of nodes in cluster.

   • clients - Number of database instances for each node.

   • disks - Number of disks in disk group.

   For example, for a four-node Oracle RAC installation, using three disks in a normal redundancy disk group, you require an additional MB of space:

   [2 * 1 * 3] + [2 * (1 * (4 * (4 + 1)+ 30)+ (64 * 4)+ 533)] = 1684 MB

   To ensure high availability of Oracle Clusterware files on Oracle ASM, you need to have at least 2 GB of disk space for Oracle Clusterware files in three separate failure groups, with at least three physical disks. Each disk must have at least 1 GB of capacity to ensure that there is sufficient space to create Oracle Clusterware files.

5. Optionally, identify failure groups for the Oracle ASM disk group devices.

   If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   Note:

   Define custom failure groups after installation, using the GUI tool ASMCA, the command line tool asmcmd, or SQL commands.

   If you define custom failure groups, then for failure groups containing database files only, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

   For failure groups containing database files and clusterware files, including voting disks, you must specify a minimum of three failure groups for normal redundancy disk groups, and five failure groups for high redundancy disk groups.

   Disk groups containing voting files must have at least 3 failure groups for normal redundancy or at least 5 failure groups for high redundancy. Otherwise, the minimum is 2 and 3 respectively. The minimum number of failure groups applies whether or not they are custom failure groups.

6. If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify multiple partitions on a single physical disk as a disk group device. Each disk group device should be on a separate physical disk.

   • Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.

     Oracle recommends that if you choose to use a logical volume manager, then use the logical volume manager to represent a single LUN without striping or mirroring, so that you can minimize the impact of the additional storage layer.

3.3.1.2 Creating Files on a NAS Device for Use with Oracle ASM

If you have a certified NAS storage device, then you can create zero-padded files in an NFS mounted directory and use those files as disk devices in an Oracle ASM disk group.

To create these files, follow these steps:

1. If necessary, create an exported directory for the disk group files on the NAS device.

   Refer to the NAS device documentation for more information about completing this step.

2. Switch user to root.

3. Create a mount point directory on the local system. For example:

   # mkdir -p /mnt/oracleasm
   

4. To ensure that the NFS file system is mounted when the system restarts, add an entry for the file system in the mount file /etc/fstab.

   See Also:

   My Oracle Support note 359515.1 for updated NAS mount option information, available at the following URL:

   https://support.oracle.com
   

   For more information about editing the mount file for the operating system, refer to the man pages. For more information about recommended mount options, refer to the section "Checking NFS Mount and Buffer Size Parameters for Oracle RAC".

5. Enter a command similar to the following to mount the NFS file system on the local system:

   # mount /mnt/oracleasm
   

6. Choose a name for the disk group to create. For example: sales1.

7. Create a directory for the files on the NFS file system, using the disk group name as the directory name. For example:

   # mkdir /mnt/oracleasm/nfsdg
   

8. Use commands similar to the following to create the required number of zero-padded files in this directory:

   # dd if=/dev/zero of=/mnt/oracleasm/nfsdg/disk1 bs=1024k count=1000
   

   This example creates 1 GB files on the NFS file system. You must create one, two, or three files respectively to create an external, normal, or high redundancy disk group.

9. Enter commands similar to the following to change the owner, group, and permissions on the directory and files that you created, where the installation owner is grid, and the OSASM group is asmadmin:

   # chown -R grid:asmadmin /mnt/oracleasm
   # chmod -R 660 /mnt/oracleasm
   

10. If you plan to install Oracle RAC or a standalone Oracle Database, then during installation, edit the Oracle ASM disk discovery string to specify a regular expression that matches the file names you created. For example:

    /mnt/oracleasm/sales1/
    

    Note:

    During installation, disk paths mounted on Oracle ASM and registered on ASMLIB with the string ORCL:* are listed as default database storage candidate disks.

3.3.1.3 Using an Existing Oracle ASM Disk Group

Select from the following choices to store either database or recovery files in an existing Oracle ASM disk group, depending on installation method:

• If you select an installation method that runs Database Configuration Assistant in interactive mode, then you can decide whether you want to create a disk group, or to use an existing one.

  The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

• If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine if an existing Oracle Automatic Storage Management disk group exists, or to determine if there is sufficient disk space in a disk group, you can use the Oracle ASM command line tool (asmcmd), Oracle Enterprise Manager Grid Control or Database Control. Alternatively, you can use the following procedure:

1. View the contents of the oratab file to determine if an Oracle Automatic Storage Management instance is configured on the system:

   $ more /etc/oratab
   

   If an Oracle Automatic Storage Management instance is configured on the system, then the oratab file should contain a line similar to the following:

   +ASM2:oracle_home_path
   

   In this example, +ASM2 is the system identifier (SID) of the Oracle Automatic Storage Management instance, with the node number appended, and oracle_home_path is the Oracle home directory where it is installed. By convention, the SID for an Oracle Automatic Storage Management instance begins with a plus sign.

2. Set the ORACLE_SID and ORACLE_HOME environment variables to specify the appropriate values for the Oracle Automatic Storage Management instance.

3. Connect to the Oracle Automatic Storage Management instance and start the instance if necessary:

   $ $ORACLE_HOME/bin/asmcmd
   ASMCMD> startup
   

4. Enter one of the following commands to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   ASMCMD> lsdb
   

   or:

   $ORACLE_HOME/bin/asmcmd -p lsdg
   

5. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

6. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   Note:

   If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

3.3.1.4 Configuring Disks for Oracle ASM with ASMLIB

The Oracle Automatic Storage Management (Oracle ASM) library driver (ASMLIB) simplifies the configuration and management of the disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.

Without ASMLIB Linux 2.6 kernel and later, block device paths do not maintain permissions and path persistence unless you create a permissions or rules file on each cluster member node; block device paths that were /dev/sda can appear as /dev/sdb after a system restart. Adding new disks requires you to modify the udev file to provide permissions and path persistence for the new disk.

With ASMLIB, you define the range of disks you want to have made available as Oracle ASM disks. ASMLIB maintains permissions and disk labels that are persistent on the storage device, so that label is available even after an operating system upgrade. You can update storage paths on all cluster member nodes by running one oracleasm command on each node.

If you intend to use Oracle ASM on block devices for database storage for Linux, then Oracle recommends that you install the ASMLIB driver and associated utilities, and use them to configure the disks for Oracle ASM.

To use the Oracle Automatic Storage Management library driver (ASMLIB) to configure Oracle ASM devices, complete the following tasks.

• Installing and Configuring the Oracle ASM Library Driver Software

• Configuring Disk Devices to Use Oracle ASM Library Driver on x86 Systems

• Configuring Disk Devices to Use ASM Library Driver on IBM zSeries Systems

• Administering the Oracle ASM Library Driver and Disks

http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

configure

# /usr/sbin/oracleasm configure -i

enable
disable

# /usr/sbin/oracleasm enable

start
stop
restart

# /usr/sbin/oracleasm restart

createdisk

# /usr/sbin/oracleasm createdisk DISKNAME devicename

deletedisk

# /usr/sbin/oracleasm deletedisk DISKNAME

querydisk

# /usr/sbin/oracleasm querydisk {DISKNAME | devicename}

listdisks

# /usr/sbin/oracleasm listdisks

scandisks

# /usr/sbin/oracleasm scandisks

3.3.1.5 Configuring ASMLIB for Multipath Disks

Additional configuration is required to use the Oracle Automatic Storage Management library Driver (ASMLIB) with third party vendor multipath disks.

https://support.oracle.com

multipaths {
       multipath {
               wwid                    3600508b4000156d700012000000b0000
               alias                   multipath
               ...
       }
       multipath {
               ...
               alias                   mympath
               ...
       }
       ...
}

3.3.1.6 Configuring Disk Devices Manually for Oracle ASM

By default, the 2.6 kernel device file naming scheme udev dynamically creates device file names when the server is started, and assigns ownership of them to root. If udev applies default settings, then it changes device file names and owners for voting disks or Oracle Cluster Registry partitions, corrupting them when the server is restarted. For example, a voting disk on a device named /dev/sdd owned by the user grid may be on a device named /dev/sdf owned by root after restarting the server.If you use ASMLIB, then you do not need to ensure permissions and device path persistency in udev.

If you do not use ASMLIB, then you must create a custom rules file. When udev is started, it sequentially carries out rules (configuration directives) defined in rules files. These files are in the path /etc/udev/rules.d/. Rules files are read in lexical order. For example, rules in the file 10-wacom.rules are parsed and carried out before rules in the rules file 90-ib.rules.

When specifying the device information in the UDEV rules file, ensure that the OWNER, GROUP and MODE are specified before any other characteristics in the order shown. For example, if you want to include the characteristic ACTION on the UDEV line, then specify ACTION after OWNER, GROUP, and MODE.

Where rules files describe the same devices, on the supported Linux kernel versions, the last file read is the one that is applied.

To configure a permissions file for disk devices, complete the following tasks:

1. To obtain information about existing block devices, run the command scsi_id (/sbin/scsi_id) on storage devices from one cluster node to obtain their unique device identifiers. When running the scsi_id command with the -s argument, the device path and name passed should be that relative to the sysfs directory /sys (for example, /block/device) when referring to /sys/block/device. For example:

   # /sbin/scsi_id -g -s /block/sdb/sdb1
   360a98000686f6959684a453333524174
    
   # /sbin/scsi_id -g -s /block/sde/sde1
   360a98000686f6959684a453333524179
   

   Record the unique SCSI identifiers of clusterware devices, so you can provide them when required.

   Note:

   The command scsi_id should return the same device identifier value for a given device, regardless of which node the command is run from.

2. Configure SCSI devices as trusted devices (white listed), by editing the /etc/scsi_id.config file and adding options=-g to the file. For example:

   # cat > /etc/scsi_id.config
   vendor="ATA",options=-p 0x80
   options=-g
   

3. Using a text editor, create a UDEV rules file for the Oracle ASM devices, setting permissions to 0660 for the installation owner and the group whose members are administrators of the Oracle Grid Infrastructure software. For example, using the installation owner grid and using a role-based group configuration, with the OSASM group asmadmin:

   # vi /etc/udev/rules.d/99-oracle-asmdevices.rules
   
   KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
   RESULT=="14f70656e66696c00000000", OWNER="grid", GROUP="asmadmin", MODE="0660"
   KERNEL=="sd?2", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
   RESULT=="14f70656e66696c00000001", OWNER="grid", GROUP="asmadmin", MODE="0660"
   KERNEL=="sd?3", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
   RESULT=="14f70656e66696c00000002", OWNER="grid", GROUP="asmadmin", MODE="0660"
   

4. Copy the rules.d file to all other nodes on the cluster. For example:

   # scp 99-oracle-asmdevices.rules root@node2:/etc/udev/rules.d/99-oracle-asmdevices.rules
   

5. Load updated block device partition tables on all member nodes of the cluster, using /sbin/partprobe devicename. You must do this as root.

6. Run the command udevtest (/sbin/udevtest) to test the UDEV rules configuration you have created. The output should indicate that the block devices are available and the rules are applied as expected. For example:

   # udevtest /block/sdb/sdb1
   main: looking at device '/block/sdb/sdb1' from subsystem 'block'
   udev_rules_get_name: add symlink
   'disk/by-id/scsi-360a98000686f6959684a453333524174-part1'
   udev_rules_get_name: add symlink
   'disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.887085-part1'
   udev_node_mknod: preserve file '/dev/.tmp-8-17', because it has correct dev_t
   run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'
   run_program: '/lib/udev/vol_id' returned with status 4
   run_program: '/sbin/scsi_id'
   run_program: '/sbin/scsi_id' (stdout) '360a98000686f6959684a453333524174'
   run_program: '/sbin/scsi_id' returned with status 0
   udev_rules_get_name: rule applied, 'sdb1' becomes 'ocr1'
   udev_device_event: device '/block/sdb/sdb1' validate currently present symlinks
   udev_node_add: creating device node '/dev/ocr1', major = '8', minor = '17', 
   mode = '0640', uid = '0', gid = '500'
   udev_node_add: creating symlink
   '/dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1' to '../../ocr1'
   udev_node_add: creating symlink
   '/dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085
   -part1' to '../../ocr1'
   main: run: 'socket:/org/kernel/udev/monitor'
   main: run: '/lib/udev/udev_run_devd'
   main: run: 'socket:/org/freedesktop/hal/udev_event'
   main: run: '/sbin/pam_console_apply /dev/ocr1
   /dev/disk/by-id/scsi-360a98000686f6959684a453333524174-part1
   /dev/disk/by-path/ip-192.168.1.1:3260-iscsi-iqn.1992-08.com.netapp:sn.84187085-
   part1'
   

   In the example output, note that applying the rules renames OCR device /dev/sdb1 to /dev/ocr1.

7. Enter the command to restart the UDEV service.

   On Asianux, Oracle Linux 5, and RHEL5, the commands are:

   # /sbin/udevcontrol reload_rules
   # /sbin/start_udev
   

   On SUSE 10 and 11, the command is:

   # /etc/init.d boot.udev restart
   

3.3.2.1 Identifying and Using Existing Oracle Database Diskgroups on Oracle ASM

The following section describes how to identify existing disk groups and determine the free disk space that they contain.

• Optionally, identify failure groups for the Oracle ASM disk group devices.

  If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

  To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

  Note:

  If you define custom failure groups, then you must specify a minimum of two failure groups for normal redundancy and three failure groups for high redundancy.

3.3.2.2 Creating Diskgroups for Oracle Database Data Files

If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

• All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

• Do not specify multiple partitions on a single physical disk as a disk group device. Oracle ASM expects each disk group device to be on a separate physical disk.

• Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.