4 Configuring Automatic Restart of an Oracle Database

Oracle Restart Overview

Oracle Restart improves the availability of your Oracle database. When you install Oracle Restart, various Oracle components can be automatically restarted after a hardware or software failure or whenever your database host computer restarts. Table 4-1 lists these components.

Oracle Restart runs periodic check operations to monitor the health of these components. If a check operation fails for a component, the component is shut down and restarted.

Oracle Restart is used in standalone server (non-clustered) environments only. For Oracle Real Application Clusters (Oracle RAC) environments, the functionality to automatically restart components is provided by Oracle Clusterware.

Oracle Restart runs out of the Oracle Grid Infrastructure home, which you install separately from Oracle Database homes. See the Oracle Database Installation Guide for your platform for information about installing the Oracle Grid Infrastructure home.

About Startup Dependencies

Oracle Restart ensures that Oracle components are started in the proper order, in accordance with component dependencies. For example, if database files are stored in Oracle ASM disk groups, then before starting the database instance, Oracle Restart ensures that the Oracle ASM instance is started and the required disk groups are mounted. Likewise, if a component must be shut down, Oracle Restart ensures that dependent components are cleanly shut down first.

Oracle Restart also manages the weak dependency between database instances and the Oracle Net listener (the listener): When a database instance is started, Oracle Restart attempts to start the listener. If the listener startup fails, then the database is still started. If the listener later fails, Oracle Restart does not shut down and restart any database instances.

About Starting and Stopping Components with Oracle Restart

Oracle Restart automatically restarts various Oracle components when required, and automatically stops Oracle components in an orderly fashion when you manually shut down your system. There may be times, however, when you want to manually start or stop individual Oracle components. Oracle Restart includes the Server Control (SRVCTL) utility that you use to manually start and stop Oracle Restart–managed components. When Oracle Restart is in use, Oracle strongly recommends that you use SRVCTL to manually start and stop components.

After you stop a component with SRVCTL, Oracle Restart does not automatically restart that component if a failure occurs. If you then start the component with SRVCTL, that component is again available for automatic restart.

Oracle utilities such as SQL*Plus, the Listener Control utility (LSNRCTL), and ASMCMD are integrated with Oracle Restart. If you shut down the database with SQL*Plus, Oracle Restart does not interpret this as a database failure and does not attempt to restart the database. Similarly, if you shut down the Oracle ASM instance with SQL*Plus or ASMCMD, Oracle Restart does not attempt to restart it.

An important difference between starting a component with SRVCTL and starting it with SQL*Plus (or another utility) is the following:

• When you start a component with SRVCTL, any components on which this component depends are automatically started first, and in the proper order.

• When you start a component with SQL*Plus (or another utility), other components in the dependency chain are not automatically started; you must ensure that any components on which this component depends are started.

In addition, Oracle Restart enables you to start and stop all of the components managed by Oracle Restart in a specified Oracle home using a single command. The Oracle home can be an Oracle Database home or an Oracle Grid Infrastructure home. This capability is useful when you are installing a patch.

About Starting and Stopping Oracle Restart

The CRSCTL utility starts and stops Oracle Restart. You can also use the CRSCTL utility to enable or disable Oracle high availability services. Oracle Restart uses Oracle high availability services to start and stop automatically the components managed by Oracle Restart. For example, Oracle high availability services daemons automatically start databases, listeners, and Oracle ASM instances. When Oracle high availability services are disabled, none of the components managed by Oracle Restart are started when a node is rebooted.

Typically, you use the CRSCTL utility when you must stop all of the running Oracle software in an Oracle installation. For example, you might need to stop Oracle Restart when you are installing a patch or performing operating system maintenance. When the maintenance is complete, you use the CRSCTL utility to start Oracle Restart.

Oracle Restart Configuration

Oracle Restart maintains a list of all the Oracle components that it manages, and maintains configuration information for each component. All of this information is collectively known as the Oracle Restart configuration. When Oracle Restart starts a component, it starts the component according to the configuration information for that component. For example, the Oracle Restart configuration includes the location of the server parameter file (SPFILE) for databases, and the TCP port to listen on for listeners.

If you install Oracle Restart and then create your database with Database Configuration Assistant (DBCA), DBCA automatically adds the database to the Oracle Restart configuration. When DBCA then starts the database, the required dependencies between the database and other components (for example disk groups in which the database stores data) are established, and Oracle Restart begins to manage the database.

You can manually add and remove components from the Oracle Restart configuration with SRVCTL commands. For example, if you install Oracle Restart onto a host on which a database is already running, you can use SRVCTL to add that database to the Oracle Restart configuration. When you manually add a component to the Oracle Restart configuration and then start it with SRVCTL, Oracle Restart begins to manage the component, restarting it when required.

Other SRVCTL commands enable you to view the status and configuration of Oracle Restart–managed components, temporarily disable and then reenable management for components, and more.

When Oracle Restart is installed, many operations that create Oracle components automatically add the components to the Oracle Restart configuration. Table 4-2 lists some create operations and whether the created component is automatically added.

^Footnote 1 Not recommended when Oracle Restart is in use

Table 4-3 lists some delete/drop/remove operations and whether the deleted component is also automatically removed from the Oracle Restart configuration.

^Footnote 1 Not recommended

Oracle Restart Integration with Oracle Data Guard

Oracle Restart is integrated with Oracle Data Guard (Data Guard) and the Oracle Data Guard Broker (the broker). When a database shutdown and restart is required in response to a role change request, Oracle Restart shuts down and restarts the database in an orderly fashion (taking dependencies into account), and according to the settings in the Oracle Restart configuration. Oracle Restart also ensures that, following a Data Guard role transition, all database services configured to run in the new database role are active and all services not configured to run in the new role are stopped.

In addition, the Oracle Restart configuration supports Data Guard–related configuration options for the following components:

• Databases—When you add a database to the Oracle Restart configuration, you can specify the current Data Guard role for the database: PRIMARY, PHYSICAL_STANDBY, LOGICAL_STANDBY, or SNAPSHOT_STANDBY. If the role is later changed using the broker, Oracle Restart automatically updates the database configuration with the new role. If you change the database role without using the broker, you must manually modify the database's role in the Oracle Restart configuration to reflect the new role.

• Database Services—When adding a database service to the Oracle Restart configuration, you can specify one or more Data Guard roles for the service. When this configuration option is present, upon database restart Oracle Restart starts the service only if one of the service roles matches the current database role.

Fast Application Notification with Oracle Restart

In a standalone server environment, Oracle Restart uses Oracle Notification Services (ONS) and Oracle Advanced Queues to publish Fast Application Notification (FAN) high availability events. Integrated Oracle clients use FAN to provide fast notification to clients when the service or instance goes down. The client can automate the failover of database connections between a primary database and a standby database.

This section describes how ONS and FAN work with Oracle Restart. It contains the following topics:

• Overview of Fast Application Notification

• Application High Availability with Services and FAN

• Managing Unplanned Outages

• Managing Planned Outages

• Fast Application Notification High Availability Events

• Using Fast Application Notification Callouts

• Oracle Clients That Are Integrated with Fast Application Notification

FAN event type: service_member
Properties: version=1.0 service=ERP database=FINPROD instance=FINPROD host=node1 status=up

#! /bin/ksh
FAN_LOGFILE= [your path name]/admin/log/`hostname`_uptime.log
echo $* "reported="`date` >> $FAN_LOGFILE &

NODE VERSION=1.0 host=sun880-2 status=nodedown reason=
timestamp=08-Oct-2004 04:02:14 reported=Fri Oct 8 04:02:14 PDT 2004

Preparing to Run SRVCTL

The tasks in the following sections require that you run the SRVCTL utility. You must ensure that you run SRVCTL from the correct Oracle home, and that you log in to the host computer with the correct user account. Table 4-6 lists the components that you can configure with SRVCTL, and for each component, lists the Oracle home from which you must run SRVCTL.

^Footnote 1 Assumes the listener was started from the Oracle Grid Infrastructure home. If you installed Oracle Restart for an existing database, the listener may have been started from the database home, in which case you start SRVCTL from the database home.

To prepare to run SRVCTL:

1. Use Table 4-6 to determine the Oracle home from which you must run SRVCTL.

2. If you intend to run a SRVCTL command that modifies the Oracle Restart configuration (add, remove, enable, disable, and so on), then do one of the following:

   • On UNIX and Linux, log in to the database host computer as the user who installed the Oracle home that you determined in Step 1.

   • On Windows, log in as an Administrator.

   Otherwise, log in to the host computer as any user.

3. Open the command window that you will use to enter the SRVCTL commands.

   To enter commands, you might need to ensure that the SRVCTL program is in your PATH environment variable. Otherwise, you can enter the absolute path to the program.

Obtaining Help for SRVCTL

Online help is available for the SRVCTL utility.

To obtain help for SRVCTL:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl
   

For more detailed help, enter the following command:

srvctl -h

For detailed help on a particular command, enter:

srvctl command -h

For example, to obtain help for the add command and the different options for each component type, enter:

srvctl add -h

For detailed help on a particular command for a particular component type, enter:

srvctl command object -h

For example, to obtain help about adding a database service, enter the following command:

srvctl add service -h

See Table 4-7 for a list of SRVCTL commands and Table 4-8 for a list of components.

Adding Components to the Oracle Restart Configuration

In most cases, creating an Oracle component on a host that is running Oracle Restart automatically adds the component to the Oracle Restart configuration. (See Table 4-2.) The component is then automatically restarted when required.

The following are occasions when you must manually add components to the Oracle Restart configuration with SRVCTL:

• You install Oracle Restart after creating the database.

• You create an additional Oracle database on the same host computer using the CREATE DATABASE SQL statement.

• You create a database service with DBMS_SERVICE.CREATE_SERVICE package procedure. (The recommended way is to use SRVCTL.)

Adding a component to the Oracle Restart configuration does not start that component. You must use a srvctl start command to start it.

You can also use Oracle Enterprise Manager Database Control (Database Control) to add a database or listener to the Oracle Restart configuration. Both the SRVCTL and Database Control methods are described in the following sections:

• Adding Components with SRVCTL

• Adding Components with Oracle Enterprise Manager Database Control

srvctl add database -d dbcrm -o /u01/app/oracle/product/11.2.0/dbhome_1 -c SINGLE

srvctl add service -d dbcrm -s crmbatch

srvctl add listener

Removing Components from the Oracle Restart Configuration

When you use an Oracle-recommended method to delete an Oracle component, the component is also automatically removed from the Oracle Restart configuration. For example, if you use Database Configuration Assistant (DBCA) to delete a database, DBCA removes the database from the Oracle Restart configuration. Likewise, if you use Oracle Net Configuration Assistant (NETCA) to delete a listener, NETCA removes the listener from the Oracle Restart configuration. See Table 4-3 for more examples. If you use a non-recommended or manual method to delete an Oracle component, you must first use SRVCTL to remove the component from the Oracle Restart configuration. Failing to do so could result in an error.

To remove a component from the Oracle Restart configuration:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl remove object [options]
   

   where object is one of the components listed in Table 4-8. See the SRVCTL remove command for available options for each component.

srvctl remove database -d dbcrm

Disabling and Enabling Oracle Restart Management for a Component

You can temporarily disable Oracle Restart management for a component. One reason to do this is when you are performing maintenance on the component. For example, if a component must be repaired, then you might not want it to be automatically restarted if it fails or if the host computer is restarted.

When maintenance is complete, you can reenable management for the component.

When you disable a component:

• It is no longer automatically restarted.

• It is no longer automatically started through a dependency.

• It cannot be started with SRVCTL.

• Any component dependent on this resource is no longer automatically started or restarted.

To disable or enable automatic restart for a component:

1. Prepare to run SRVCTL, as described in "Preparing to Run SRVCTL".

2. Do one of the following:

   • To disable a component, enter the following command:

     srvctl disable object [options]
     

   • To enable a component, enter the following command:

     srvctl enable object [options]
     

   where object is one of the components listed in Table 4-8. See the SRVCTL disable command and the enable command for available options for each component.

srvctl disable database -d dbcrm

srvctl disable diskgroup -g recovery

srvctl enable diskgroup -g recovery

Viewing Component Status

You can use SRVCTL to view the running status (running or not running) for any component managed by Oracle Restart. For some components, additional information is also displayed.

To view component status:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl status object [options]
   

   where object is one of the components listed in Table 4-8. See the SRVCTL status command for available options for each component.

srvctl status database -d dbcrm

Database is running.

Viewing the Oracle Restart Configuration for a Component

You can use SRVCTL to view the Oracle Restart configuration for any component. Oracle Restart maintains different configuration information for each component type. In one form of the SRVCTL command, you can obtain a list of components managed by Oracle Restart.

To view component configuration:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl config object options
   

   where object is one of the components listed in Table 4-8. See the SRVCTL config command for available options for each component.

srvctl config database

dbcrm
orcl

srvctl config database -d orcl

Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/orcl/spfileorcl.ora
Domain: us.example.com
Start options: open
Stop options: immediate
Database role:
Management policy: automatic
Disk Groups: DATA
Services: mfg,sales

Modifying the Oracle Restart Configuration for a Component

You can use SRVCTL to modify the Oracle Restart configuration of a component. For example, you can modify the port number that a listener listens on when Oracle Restart starts it, or the server parameter file (SPFILE) that Oracle Restart points to when it starts a database.

To modify the Oracle Restart configuration for a component:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl modify object options
   

   where object is one of the components listed in Table 4-8. See the SRVCTL modify command for available options for each component.

srvctl modify database -d dbcrm -y MANUAL -s NOMOUNT

With a MANUAL management policy, the database is never automatically started when the database host computer is restarted. However, Oracle Restart continues to monitor the database and restarts it if a failure occurs.

Managing Environment Variables in the Oracle Restart Configuration

The Oracle Restart configuration can store name/value pairs for environment variables. If you typically set environment variables (other than ORACLE_HOME and ORACLE_SID) prior to starting your Oracle database, you can set these environment variable values in the Oracle Restart configuration. You can store any number environment variables in the individual configurations of the following components:

• Database instance

• Listener

• Oracle ASM instance

When Oracle Restart starts one of these components, it first sets environment variables for that component to the values stored in the component configuration. Although you can set environment variables that are used by Oracle components in this manner, this capability is primarily intended for operating system environment variables.

The following sections provide instructions for setting, unsetting, and viewing environment variables:

• Setting and Unsetting Environment Variables

• Viewing Environment Variables

srvctl setenv database -d dbcrm -t "NLS_LANG=AMERICAN_AMERICA.AL32UTF8, 
   AIXTHREAD_SCOPE=S"

srvctl getenv database -d dbcrm

dbcrm:
NLS_LANG=AMERICAN_AMERICA
AIXTHREAD_SCOPE=S
GCONF_LOCAL_LOCKS=1

srvctl getenv database -d dbcrm -t "NLS_LANG,AIXTHREAD_SCOPE"

dbcrm:
NLS_LANG=AMERICAN_AMERICA
AIXTHREAD_SCOPE=S

Creating and Deleting Database Services with SRVCTL

When managing a database with Oracle Restart, Oracle recommends that you use SRVCTL to create and delete database services. When you use SRVCTL to add a database service, the service is automatically added to the Oracle Restart configuration and a dependency between the service and the database is established. Thus, if you start the service, Oracle Restart first starts the database if it is not started.

When you use SRVCTL to delete a database service, the service is also removed from the Oracle Restart configuration.

To create a database service with SRVCTL:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl add service -d db_unique_name -s service_name [options]
   

   The database service is created and added to the Oracle Restart configuration. See the srvctl add service command for available options.

srvctl add service -d dbcrm -s crmbatch

srvctl add service -d dbcrm -s crmbatch -l PHYSICAL_STANDBY

To delete a database service with SRVCTL:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Enter the following command:

   srvctl remove service -d db_unique_name -s service_name [-f]
   

   The database service is removed from the Oracle Restart configuration. If the -f (force) flag is present, the service is removed even if it is still running. Without this flag, an error occurs if the service is running.

Enabling FAN Events in an Oracle Restart Environment

To enable Oracle Restart to publish Fast Application Notification (FAN) events, you must create an Oracle Notification Services (ONS) network that includes the Oracle Restart servers and the integrated clients. These clients can include Oracle Connection Manager (CMAN), Java Database Connectivity (JDBC), and Universal Connection Pool (UCP) clients. If you are using Oracle Call Interface or ODP.NET clients, then you must enable Oracle Advanced Queuing (AQ) HA notifications for your services. In addition, ONS must be running on the server.

To enable FAN events in an Oracle Restart environment:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Add the database to the Oracle Restart Configuration if it is not already managed by Oracle Restart. See "Adding Components to the Oracle Restart Configuration".

3. Add ONS to the configuration:

   srvctl add ons
   

   ONS is disabled when it is added.

4. Enable ONS:

   srvctl enable ons
   

5. Start ONS:

   srvctl start ons
   

6. Add the service to the Oracle Restart Configuration.

   For Oracle Call Interface and ODP.NET clients, ensure that the -q option is set to TRUE to enable the database queue.

   See "Creating and Deleting Database Services with SRVCTL".

7. Enable each client for fast connection failover. See "Enabling Clients for Fast Connection Failover".

Automating the Failover of Connections Between Primary and Standby Databases

In a configuration that uses Oracle Restart and Oracle Data Guard primary and standby databases, the database services fail over automatically from the primary to the standby during either a switchover or failover. You can use Oracle Notification Services (ONS) to immediately notify clients of the failover of services between the primary and standby databases. The Oracle Data Guard Broker uses Fast Application Notification (FAN) to send notifications to clients when a failover occurs. Integrated Oracle clients automatically failover connections and applications can mask the failure from end-users.

To automate connection failover, you must create an ONS network that includes the Oracle Restart servers and the integrated clients (CMAN, listener, JDBC, and UCP). If you are using Oracle Call Interface or ODP.NET clients, you must enable the Oracle Advanced Queuing queue. The database and the services must be managed by Oracle Restart and the Oracle Data Guard Broker to automate the failover of services.

To automate the failover of services between primary and standby databases:

1. Configure the primary and standby database with the Oracle Data Guard Broker. See Oracle Data Guard Broker.

2. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

3. Add the primary database to the Oracle Restart configuration on the primary server if it has not been added. Ensure that you specify PRIMARY for the database role. See "Adding Components to the Oracle Restart Configuration".

4. Add the standby database to the Oracle Restart configuration on the standby server if it has not been added. Ensure that you specify the appropriate standby database role.

5. Enable FAN events on both the primary database server and the standby database server. "Enabling FAN Events in an Oracle Restart Environment".

6. Add the services that clients will use to connect to the databases to the Oracle Restart configuration on the primary database and the standby database. When you add a service, ensure that:

   • The -l option is set to the proper role for each service

   • The -q option is set to TRUE if you are using ODP.NET or Oracle Call Interface

   See "Creating and Deleting Database Services with SRVCTL".

7. Enable each client for fast connection failover. See "Enabling Clients for Fast Connection Failover".

Enabling Clients for Fast Connection Failover

In a configuration with a standby database, after you have added Oracle Notification Services (ONS) to your Oracle Restart configurations and enabled Oracle Advanced Queuing (AQ) HA notifications for your services, you can enable clients for fast connection failover. The clients receive Fast Application Notification (FAN) events and can relocate connections to the current primary database after an Oracle Data Guard failover. See "Automating the Failover of Connections Between Primary and Standby Databases" for information about adding ONS.

For databases with no standby database configured, you can still configure the client FAN events. When there is a failure, you can configure the client to retry the connection to the database. Since Oracle Restart will restart the failed database, the client can reconnect when the database restarts. Ensure that you program the appropriate delay and retries on the connection string, as illustrated in the examples in this section.

You can enable fast connection failover for the following types of clients in an Oracle Restart configuration:

• Enabling Fast Connection Failover for JDBC Clients

• Enabling Fast Connection Failover for Oracle Call Interface Clients

• Enabling Fast Connection Failover for ODP.NET Clients

Starting and Stopping Components Managed by Oracle Restart with SRVCTL

You can start and stop any component managed by Oracle Restart with SRVCTL.

To start or stop a component managed by Oracle Restart with SRVCTL:

1. Prepare to run SRVCTL as described in "Preparing to Run SRVCTL".

2. Do one of the following:

   • To start a component, enter the following command:

     srvctl start object [options]
     

   • To stop a component, enter the following command:

     srvctl stop object [options]
     

   where object is one of the components listed in Table 4-8. See the SRVCTL start command and the stop command for available options for each component.

srvctl start database -d dbcrm

srvctl start database -d dbcrm -o nomount

srvctl start listener

srvctl start listener -l crmlistener

srvctl start service -d dbcrm -s "bizdev,support"

srvctl start diskgroup -g "data,recovery"

srvctl stop database -d dbcrm

srvctl stop database -d dbcrm -o abort

Starting a Database Managed by Oracle Restart with Oracle Enterprise Manager

With Oracle Enterprise Manager Database Control (Database Control), you can use Oracle Restart to start a database.

To start a database managed by Oracle Restart with Oracle Enterprise Manager:

1. Access the Database Home page for the desired database instance.

   See "Accessing the Database Home Page" in Oracle Database 2 Day DBA for instructions.

2. Click Startup.

   The Startup/Shutdown Credentials page appears.

3. Enter credentials as follows:

   1. Enter the host computer credentials for the user who installed the database Oracle home.

   2. Enter the database credentials consisting of the user name SYS and the password that you assigned to SYS during the installation.

   3. In the Connect As list, choose the value SYSOPER.

4. (Optional) Select the Save as Preferred Credential option if you want these credentials to be automatically filled in for you the next time that this page appears.

5. Click OK.

   The Select Startup Type page appears.

6. To start the database with Oracle Restart, select Start database along with dependent resources.

   This ensures that resources on which the database depends, such as the Oracle Automatic Storage Management instance, are successfully started before the database is started.

7. Click OK.

   A confirmation page appears.

8. Click Yes.

   The Startup/Shutdown: Activity Information page appears, indicating that the database is being started up. When startup is complete, the Login page appears.

9. Log in to the database (and to Database Control).

   The Database Home page appears indicating that the database instance status is Up.