The number of resources you register with Oracle Clusterware to manage an application depends on the application. Applications that consist of only one process are usually represented by only one resource. For more complex applications that are built on multiple processes or components that may require multiple resources, you can create resource groups.

Each resource is based on a resource type that serves as a template. You can configure how Oracle Clusterware will place an application in the cluster by specifying an explicit list of servers, or by using features such as server pools and policies. Relationships between applications and components are expressed using dependencies. Oracle Clusterware manages the application by performing operations on the resources, and the resource state represents the availability of the application.

When you register an application as a resource in Oracle Clusterware, in addition to actually adding the resource to the system, you define how Oracle Clusterware manages the application using resource attributes you ascribe to the resource. The frequency with which the resource is checked and the number of attempts to restart a resource on the same server after a failure before attempting to start it on another server (failover) are examples of resource attributes. The registration information also includes a path to an action script or application-specific action program that Oracle Clusterware calls to start, stop, check, and clean up the application.

An action script is a shell script (a batch script in Windows) that a generic script agent provided by Oracle Clusterware calls. An application-specific agent is usually a C or C++ program that calls Oracle Clusterware-provided APIs directly.

Critical Resources

Some large enterprise applications modeled as resource groups can comprise multiple resources representing application or infrastructure components. If any resource in the resource group fails, then Oracle Clusterware must fail the entire resource group over to another server in the cluster.

You can mark a resource as critical for its resource group by specifying the name of the resource in the CRITICAL_RESOURCES attribute of the resource group.

Generally, all resources are unique but some resources may have common attributes. Oracle Clusterware uses resource types to organize these similar resources. Benefits that resource types provide are:

• Manage only necessary resource attributes

• Manage all resources based on the resource type

Every resource that you register in Oracle Clusterware must have a certain resource type. In addition to the resource types included in Oracle Clusterware, you can define custom resource types using the Oracle Clusterware Control (CRSCTL) utility. The included resource types are:

• Local resource: Instances of local resources—type name is local_resource—run on each server of the cluster (the default) or you can limit them to run on servers belonging to a particular server category. When a server joins the cluster, Oracle Clusterware automatically extends local resources to have instances tied to the new server. When a server leaves the cluster, Oracle Clusterware automatically sheds the instances of local resources that ran on the departing server. Instances of local resources are pinned to their servers; they do not fail over from one server to another.

• Cluster resource: Cluster-aware resource types—type name is cluster_resource—are aware of the cluster environment and are subject to cardinality and cross-server switchover and failover.

• Generic application: You can use this resource type—type name is generic_application—to protect any generic applications without requiring additional scripts. High availability for an application is achieved by defining a resource with the generic_application resource type and providing the values for key attributes of the resource. The generic_application resource type is derived from the cluster_resource resource type and, therefore, all resources of the generic_application resource type are cluster-aware resources. Attributes include:

  • START_PROGRAM: A complete path to the executable that starts the application, with all appropriate arguments. The executable must exist on every server where Oracle Grid Infrastructure is configured to run the application. This attribute is required. For example:

    /opt/my_app –start

    The executable must also ensure that the application starts and return an exit status value of zero (0) to indicate that the application started successfully and is online. If the executable fails to start the application, then the executable exits with a non-zero status code.

  • STOP_PROGRAM: A complete path to the executable that stops the application, with all appropriate arguments. The executable must exist on every server where Oracle Grid Infrastructure is configured to run the application. If you do not specify this attribute value, then Oracle Clusterware uses an operating system-equivalent of the kill command. For example:

    /opt/my_app –stop

    The executable must also ensure that the application stops and return an exit status value of zero (0) to indicate that the application stopped successfully. If the executable fails to stop the application, then the executable exits with a non-zero status code and Oracle Clusterware initiates a clean of the resource.

  • CLEAN_PROGRAM: A complete path to the executable that cleans the program, with all appropriate arguments. The executable must exist on every server where Oracle Grid Infrastructure is configured to run the application. If you do not specify a value for this attribute, then Oracle Clusterware uses an operating system-equivalent of the kill -9 command. For example:

    /opt/my_app –clean

    Note:

    The difference between STOP_PROGRAM and CLEAN_PROGRAM is that CLEAN_PROGRAM is a forced stop that stops an application ungracefully, and must always be able to stop an application or the application becomes unmanageable.

  • PID_FILES: A comma-delimited list of complete paths to files that will be written by the application and contain a process ID (PID) to monitor. Failure of a single process is treated as a complete resource failure. For example:

    /opt/app.pid

    Note:

    The files that you specify in the PID_FILES attribute are read immediately after the START action completes and monitoring commences for the PIDs listed in the files.

  • EXECUTABLE_NAMES: A comma-delimited list of names of executables that is created when the application starts and the state of these executables is subsequently monitored. Failure of a single executable is treated as a complete resource failure. For example:

    my_app

    Note:

    You need specify only the complete name of the executables. This attribute does not accept the path of the executable or wild cards. The PIDs matching the executable names are cached immediately after the START action completes.

  • CHECK_PROGRAMS: A list of complete paths to the executable that determines the state of the application. Reporting a non-running state by any of the applications is treated as a failure of the entire resource. For example:

    /opt/my_app –check

  • ENVIRONMENT_FILE: A complete path to the file containing environment variables to source when starting the application. The file must be a text file containing name=value pairs, one per line. For example:

    /opt/my_app.env

  • ENVIRONMENT_VARS: A comma-delimited list of name=name pairs to be included into the environment when starting an application. For example:

    USE_FILES=No, AUTO_START=Yes

  • SEND_OUTPUT_ALWAYS: This attribute is responsible for sending the application output that is sent to STDOUT, which is then displayed. A value of 0 does not display any application output unless an action fails. When an action fails, whatever application output that has been saved by the agent is displayed. Any value greater than 0 displays every application output. The default value is 0. For example:

    SEND_OUTPUT_ALWAYS=1

  Note:

  If you do not specify the STOP_PROGRAM, CHECK_PROGRAMS, and CLEAN_PROGRAM attributes, then you must specify either PID_FILES or EXECUTABLE_NAMES, or Oracle Clusterware will not allow you to register a resource of this type.

  If you specify all the attributes, then the following rules apply:

  1. When stopping a resource, if you specified STOP_PROGRAM, then Oracle Clusterware calls STOP_PROGRAM. Otherwise, Oracle Clusterware uses an operating system-equivalent of the kill -9 command on the PID obtained from either the PID_FILES or the EXECUTABLE_NAMES attribute.

  2. When you need to establish the current state of an application, if you specified CHECK_PROGRAMS, then Oracle Clusterware calls CHECK_PROGRAMS. Otherwise, Oracle Clusterware uses an operating system-equivalent of the ps -p command with the PID obtained from either the PID_FILES or EXECUTABLE_NAMES attribute.

  3. When cleaning a resource, if you specified CLEAN_PROGRAM, then Oracle Clusterware calls CLEAN_PROGRAM. Otherwise, Oracle Clusterware uses an operating system-equivalent of the kill -9 command on the PID obtained from either the PID_FILES or the EXECUTABLE_NAMES attribute.

An agent is a process that contains the agent framework and user code to manage resources. The agent framework is a library that enables you to plug in your application-specific code to manage customized applications. You program all of the actual application management functions, such as starting, stopping and checking the health of an application, into the agent. These functions are referred to as entry points.

The agent framework is responsible for invoking these entry point functions on behalf of Oracle Clusterware. Agent developers can use these entry points to plug in the required functionality for a specific resource regarding how to start, stop, and monitor a resource. Agents are capable of managing multiple resources.

Agent developers can set the following entry points as callbacks to their code:

• ABORT: If any of the other entry points hang, the agent framework calls the ABORT entry point to stop the ongoing action. If the agent developer does not supply a stop function, then the agent framework exits the agent program.

• ACTION: The ACTION entry point is Invoked when a custom action is invoked using the clscrs_request_action API of the crsctl request action command.

• CHECK: The CHECK (monitor) entry point acts to monitor the health of a resource. The agent framework periodically calls this entry point. If it notices any state change during this action, then the agent framework notifies Oracle Clusterware about the change in the state of the specific resource.

• CLEAN: The CLEAN entry point acts whenever there is a need to clean up a resource. It is a non-graceful operation that is invoked when users must forcefully terminate a resource. This command cleans up the resource-specific environment so that the resource can be restarted.

• DELETE: The DELETE entry point is invoked on every node where a resource can run when the resource is unregistered.

• MODIFY: The MODIFY entry point is invoked on every node where a resource can run when the resource profile is modified.

• START: The START entry point acts to bring a resource online. The agent framework calls this entry point whenever it receives the start command from Oracle Clusterware.

• STOP: The STOP entry points acts to gracefully bring down a resource. The agent framework calls this entry point whenever it receives the stop command from Oracle Clusterware.

START, STOP, CHECK, and CLEAN are mandatory entry points and the agent developer must provide these entry points when building an agent. Agent developers have several options to implement these entry points, including using C, C++, or scripts. It is also possible to develop agents that use both C or C++ and script-type entry points. When initializing the agent framework, if any of the mandatory entry points are not provided, then the agent framework invokes a script pointed to by the ACTION_SCRIPT resource attribute.

At any given time, the agent framework invokes only one entry point per application. If that entry point hangs, then the agent framework calls the ABORT entry point to end the current operation. The agent framework periodically invokes the CHECK entry point to determine the state of the resource. This entry point must return one of the following states as the resource state:

• CLSAGFW_ONLINE: The CHECK entry point returns ONLINE if the resource was brought up successfully and is currently in a functioning state. The agent framework continues to monitor the resource when it is in this state. This state has a numeric value of 0 for the scriptagent.

• CLSAGFW_UNPLANNED_OFFLINE and CLSAGFW_PLANNED_OFFLINE: The OFFLINE state indicates that the resource is not currently running. These two states have numeric values of 1 and 2, respectively, for the scriptagent.

  Two distinct categories exist to describe an resource's offline state: planned and unplanned.

  When the state of the resource transitions to OFFLINE through Oracle Clusterware, then it is assumed that the intent for this resource is to be offline (TARGET=OFFLINE), regardless of which value is returned from the CHECK entry point. However, when an agent detects that the state of a resource has changed independent of Oracle Clusterware (such as somebody stopping the resource through a non-Oracle interface), then the intent must be carried over from the agent to the Cluster Ready Services daemon (CRSD). The intent then becomes the determining factor for the following:

  • Whether to keep or to change the value of the resource's TARGET resource attribute. PLANNED_OFFLINE indicates that the TARGET resource attribute must be changed to OFFLINE only if the resource was running before. If the resource was not running (STATE=OFFLINE, TARGET=OFFLINE) and a request comes in to start it, then the value of the TARGET resource attribute changes to ONLINE. The start request then goes to the agent and the agent reports back to Oracle Clusterware a PLANNED_OFFLINE resource state, and the value of the TARGET resource attribute remains ONLINE. UNPLANNED_OFFLINE does not change the TARGET attribute.

  • Whether to leave the resource's state as UNPLANNED_OFFLINE or attempt to recover the resource by restarting it locally or failing it over to a another server in the cluster. The PLANNED_OFFLINE state makes CRSD leave the resource as is, whereas the UNPLANNED_OFFLINE state prompts resource recovery.

• CLSAGFW_UNKNOWN: The CHECK entry point returns UNKNOWN if the current state of the resource cannot be determined. In response to this state, Oracle Clusterware does not attempt to failover or to restart the resource. The agent framework continues to monitor the resource if the previous state of the resource was either ONLINE or PARTIAL. This state has a numeric value of 3 for the scriptagent.

• CLSAGFW_PARTIAL: The CHECK entry point returns PARTIAL when it knows that a resource is partially ONLINE and some of its services are available. Oracle Clusterware considers this state as partially ONLINE and does not attempt to failover or to restart the resource. The agent framework continues to monitor the resource in this state. This state has a numeric value of 4 for the scriptagent.

• CLSAGFW_FAILED: The CHECK entry point returns FAILED whenever it detects that a resource is not in a functioning state and some of its components have failed and some clean up is required to restart the resource. In response to this state, Oracle Clusterware calls the CLEAN action to clean up the resource. After the CLEAN action finishes, the state of the resource is expected to be OFFLINE. Next, depending on the policy of the resource, Oracle Clusterware may attempt to failover or restart the resource. Under no circumstances does the agent framework monitor failed resources. This state has a numeric value of 5 for the scriptagent.

The agent framework implicitly monitors resources in the states listed in Table 9-2 at regular intervals, as specified by the CHECK_INTERVAL or OFFLINE_CHECK_INTERVAL resource attributes.

Whenever an agent starts, the state of all the resources it monitors is set to UNKNOWN. After receiving an initial probe request from Oracle Clusterware, the agent framework executes the CHECK entry point for all of the resources to determine their current states.

Once the CHECK action successfully completes for a resource, the state of the resource transitions to one of the previously mentioned states. The agent framework then starts resources based on commands issued from Oracle Clusterware. After the completion of every action, the agent framework invokes the CHECK action to determine the current resource state. If the resource is in one of the monitored states listed in Table 9-2, then the agent framework periodically executes the CHECK entry point to check for changes in resource state.

By default, the agent framework does not monitor resources that are offline. However, if the value of the OFFLINE_CHECK_INTERVAL attribute is greater than 0, then the agent framework monitors offline resources.

The agent framework invokes these actions without the C/C++ actions. Using action scripts, you can build an agent that contains the C/C++ entry points and the script entry points. If all of the actions are defined in the action script, then you can use the script agent to invoke the actions defined in any action scripts.

Before invoking the action defined in the action script, the agent framework exports all the necessary attributes from the resource profile to the environment. Action scripts can log messages to the stdout/stderr, and the agent framework prints those messages in the agent logs. However, action scripts can use special tags to send the progress, warning, or error messages to the crs* client tools by prefixing one of the following tags to the messages printed to stdout/stderr:

CRS_WARNING:
CRS_ERROR:
CRS_PROGRESS:

The agent framework strips out the prefixed tag when it sends the final message to the crs* clients.

Resource attributes can be accessed from within an action script as environment variables prefixed with _CRS_. For example, the START_TIMEOUT attribute becomes an environment variable named _CRS_START_TIMEOUT.

Building an agent for a specific application involves the following steps:

To register an application as a resource:

$ crsctl add resource resource_name -type [-group group_name] resource_type
  [-file file_path] | [-attr "attribute_name='attribute_value', attribute_name='
  attribute_value', ..."]

Choose a name for the resource based on the application for which it is being created. For example, if you create a resource for an Apache Web server, then you might name the resource myApache. Specify the name of an existing resource type after the -type option. Optionally, you can add the resource to an existing resource group.

You can specify resource attributes in either a text file specified with the -file option or in a comma-delimited list of resource attribute-value pairs enclosed in double quotation marks ("") following the -attr option. You must enclose space- or comma-delimited attribute names and values enclosed in parentheses in single quotation marks ('').

The following is an example of an attribute file:

PLACEMENT=favored
HOSTING_MEMBERS=node1 node2 node3
RESTART_ATTEMPTS@CARDINALITYID(1)=0
RESTART_ATTEMPTS@CARDINALITYID(2)=0
FAILURE_THRESHOLD@CARDINALITYID(1)=2
FAILURE_THRESHOLD@CARDINALITYID(2)=4
FAILURE_INTERVAL@CARDINALITYID(1)=300
FAILURE_INTERVAL@CARDINALITYID(2)=500
CHECK_INTERVAL=2
CARDINALITY=2

The following is an example of using the -attr option:

$ crsctl add resource resource_name -type resource_type [-attr "PLACEMENT='
  favored', HOSTING_MEMBERS='node1 node2 node3', ..."]

Every resource in a cluster is in a particular state at any time. Certain actions or events can cause that state to change.

Table 9-3 lists and describes the possible resource states.

If clients of an application access the application through a network, and the placement policy for the application allows it to fail over to another node, then you must register a virtual internet protocol address (VIP) on which the application depends. You should base any new application VIPs on this VIP type to ensure that your system experiences consistent behavior among all of the VIPs that you deploy in your cluster.

While you can add a VIP in the same way that you can add any other resource that Oracle Clusterware manages, Oracle recommends using the Grid_home/bin/appvipcfg command-line utility to create or delete an application VIP on the default network for which the ora.net1.network resource is created by default.

To create an application VIP, use the following syntax:

appvipcfg create -network=network_nummber -ip=ip_address -vipname=vip_name
-user=user_name [-group=group_name] [-failback=0 | 1]

When you create an application VIP on a default network, set -network=1.

To create an application VIP on a non-default network, you may have to first create the network using the srvctl add network command. Then you can create the application VIP, setting -network=non-default_network_number.

In an Oracle Flex Cluster, you can also add a non-Hub Node network resource for application VIPs, so that applications can run on non-Hub Nodes using the srvctl add network command, as follows:

srvctl add network -netnum=network_number -subnet subnet/netmask[/if1[|if2|...]]

To delete an application VIP, use the following syntax:

appvipcfg delete -vipname=vip_name

In the preceding syntax examples, network_number is the number of the network, ip_address is the IP address, vip_name is the name of the VIP, user_name is the name of the user who installed Oracle Database, and group_name is the name of the group. The default value of the -failback option is 0. If you set the option to 1, then the VIP (and therefore any resources that depend on VIP) fails back to the original node when it becomes available again.

For example, as root, run the following command:

# Grid_home/bin/appvipcfg create -network=1 -ip=148.87.58.196 -vipname=appsVIP -user=root

The script only requires a network number, the IP address, and a name for the VIP resource, in addition to the user that owns the application VIP resource. A VIP resource is typically owned by root because VIP related operations require root privileges.

To delete an application VIP, use the same script with the delete option. This option accepts the VIP name as a parameter. For example:

# Grid_home/bin/appvipcfg delete -vipname=appsVIP

After you have created the application VIP using this configuration script, you can view the VIP profile using the following command:

$ Grid_home/bin/crsctl status res appsVIP -p

Verify and, if required, modify the following parameters using the Grid_home/bin/crsctl modify res command.

The appvipcfg script requires that you specify the -network option, even if -network=1.

As the Oracle Database installation owner, start the VIP resource:

$ crsctl start resource appsVIP

To add resources to Oracle Clusterware using Oracle Enterprise Manager:

Oracle Clusterware manages resources based on the permissions of the user who added the resource. The user who first added the resource owns the resource and the resource runs as the resource owner. Certain resources must be managed as root. If a user other than root adds a resource that must be run as root, then the permissions must be changed as root so that root manages the resource, as follows:

The PLACEMENT resource attribute determines how Oracle Clusterware selects a server on which to start a resource and where to relocate the resource after a server failure. The HOSTING_MEMBERS and SERVER_POOLS attributes determine eligible servers to host a resource and the PLACEMENT attribute further refines the placement of resources.

The value of the PLACEMENT resource attribute determines how Oracle Clusterware places resources when they are added to the cluster or when a server fails. Together with either the HOSTING_MEMBERS or SERVER_POOLS attributes, you can configure how Oracle Clusterware places the resources in a cluster. When the value of the PLACEMENT attribute is:

• balanced: Oracle Clusterware uses any online server pool for placement. Less loaded servers are preferred to servers with greater loads. To measure how loaded a server is, Oracle Clusterware uses the LOAD resource attribute of the resources that are in an ONLINE state on the server. Oracle Clusterware uses the sum total of the LOAD values to measure the current server load.

• favored: If a value is assigned to either of the HOSTING_MEMBERS, SERVER_POOLS, or SERVER_CATEGORY resource attributes, then that value expresses a preference. If HOSTING_MEMBERS is populated and either SERVER_POOLS or SERVER_CATEGORY is set, then HOSTING_MEMBERS indicates placement preference and SERVER_POOLS or SERVER_CATEGORY indicates a restriction. For example, the ora.cluster.vip resource has a policy that sets the value of PLACEMENT to favored, SERVER_CATEGORY is set to Hub, and HOSTING_MEMBERS is set to server_name1. In this case, Oracle Clusterware restricts the placement of ora.cluster.vip to the servers in the Hub category and then it prefers the server known as server_name1.

• restricted: Oracle Clusterware only considers servers that belong to server pools listed in the SEVER_POOLS resource attribute, servers of a particular category as configured in the SERVER_CATEGORY resource attribute, or servers listed in the HOSTING_MEMBERS resource attribute for resource placement. Only one of these resource attributes can have a value, otherwise it results in an error.

To unregister a resource, use the crsctl delete resource command. You cannot unregister an application or resource that is ONLINE or required by another resource, unless you use the -force option. The following example unregisters the Apache Web server application:

$ crsctl delete resource myApache

Run the crsctl delete resource command as a clean-up step when a resource is no longer managed by Oracle Clusterware. Oracle recommends that you unregister any unnecessary resources.

Use the crsctl add resource command to register applications in OCR. For example, enter the following command to register the Apache Web server application from the previous example:

$ crsctl add resource myApache -type cluster_resource
-group group_name -attr "ACTION_SCRIPT=/opt/cluster/scripts/myapache.scr, PLACEMENT=restricted,
SERVER_POOLS=server_pool_list,CHECK_INTERVAL=30,RESTART_ATTEMPTS=2,
START_DEPENDENCIES=hard(appsvip),STOP_DEPENDENCIES=hard(appsvip)"

In the preceding example, you can assign the resource to a resource group by specifying the -group parameter.

If you modify a resource, then update OCR by running the crsctl modify resource command.

Manually starting or stopping resources outside of Oracle Clusterware can invalidate the resource status. In addition, Oracle Clusterware may attempt to restart a resource on which you perform a manual stop operation.

To start an application resource that is registered with Oracle Clusterware, use the crsctl start resource command. For example:

$ crsctl start resource myApache

The command waits to receive a notification of success or failure from the action program each time the action program is called. Oracle Clusterware can start application resources if they have stopped due to exceeding their failure threshold values. You must register a resource using crsctl add resource before you can start it.

Running the crsctl start resource command on a resource sets the resource TARGET value to ONLINE. Oracle Clusterware attempts to change the state to match the TARGET by running the action program with the start action.

If a cluster server fails while you are starting a resource on that server, then check the state of the resource on the cluster by using the crsctl status resource command.

For example, to relocate the Apache Web server application to a server named rac2, run the following command:

# crsctl relocate resource myApache -n rac2

Each time that the action program is called, the crsctl relocate resource command waits for the duration specified by the value of the SCRIPT_TIMEOUT resource attribute to receive notification of success or failure from the action program. A relocation attempt fails if:

• The application has required resources that run on the initial server

• Applications that require the specified resource run on the initial server

To relocate an application and its required resources, use the -f option with the crsctl relocate resource command. Oracle Clusterware relocates or starts all resources that are required by the application regardless of their state.

You can also relocate a resource group using the crsctl relocate resourcegroup command, which first stops the resources in the resource group before relocating the resource group on the destination server.

Online Relocation

The command sets the resource TARGET value to OFFLINE. Because Oracle Clusterware always attempts to match the state of a resource to its target, the Oracle Clusterware subsystem stops the application. The following example stops the Apache Web server:

# crsctl stop resource myApache

You cannot stop a resource if another resource has a hard stop dependency on it, unless you use the force (-f) option. If you use the crsctl stop resource resource_name -f command on a resource upon which other resources depend, and if those resources are running, then Oracle Clusterware stops the resource and all of the resources that depend on the resource that you are stopping.

The following example displays the status information for the Apache Web server application:

# crsctl status resource myApache

NAME=myApache
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on server010

Other information this command returns includes the following:

• How many times the resource has been restarted

• How many times the resource has failed within the failure interval

• The maximum number of times that a resource can restart or fail

• The target state of the resource and the normal status information

Use the -f option with the crsctl status resource resource_name command to view full information of a specific resource.

Enter the following command to view information about all applications and resources in tabular format:

# crsctl status resource -t

When a server restarts, Oracle Clusterware attempts to start the resources that run on the server as soon as the server starts. Resource startup might fail, however, if system components on which a resource depends, such as a volume manager or a file system, are not running. This is especially true if Oracle Clusterware does not manage the system components on which a resource depends.