1 Introduction to Installing and Configuring Oracle Clusterware and Oracle Real Application Clusters

1.1.1 Oracle Real Application Clusters Administration and Deployment Guide

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide describes how to administer Oracle Clusterware components such as the voting disks and Oracle Cluster Registry (OCR) devices. This book also explains how to administer storage, how to use RAC scalability features to add and delete instances and nodes, how to use Recovery Manager (RMAN), and how to perform backup and recovery in RAC.

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide describes RAC deployment topics such as services, high availability, and workload management. The book describes how the Automatic Workload Repository (AWR) tracks and reports service levels, and how you can use service level thresholds and alerts to improve high availability in your RAC environment. The book also describes how to make your applications highly available using Oracle Clusterware.

Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide also provides information about how to monitor and tune performance in RAC environments by using Oracle Enterprise Manager, and by using information in AWR and Oracle Database performance views. This book also provides some application-specific deployment techniques for online transaction processing and data warehousing environments.

1.2.1 Overview of Cluster Verification Utility

Cluster Verification Utility (CVU) is provided with Oracle Clusterware and Oracle Database 10g Release 2 (10.2) with Real Application Clusters. The purpose of CVU is to enable you or your hardware vendors to verify during setup and configuration that all components required for a successful installation of Oracle Clusterware or Oracle Clusterware and a RAC database are installed and configured correctly, and to provide you with ongoing assistance any time you need to make changes to your RAC cluster. You are provided with commands to use CVU to verify completion of tasks in this guide.

There are two types of CVU commands:

• Stage Commands are CVU commands used to test system setup and readiness for successful software installation, database creation, or configuration change steps. These commands are also used to validate successful completion of specific cluster configuration steps.

• Component Commands are CVU commands used to check individual cluster components, and determine their state.

This guide provides stage and component CVU commands where appropriate to assist you with cluster verification.

1.2.2 Overview of Oracle Universal Installer

Oracle Universal Installer (OUI) is a graphical user interface (GUI) tool that assists you with installing and configuring Oracle Database. It can be run using different command options to perform installation pre-checks, specialized installation processes, and other tasks. To see an overview of OUI options, navigate to the directory path oui/bin in the Oracle home directory, and type the following command:

$ ./runInstaller -help

1.2.3 Overview of Upgrading and Multiple-release Systems

The path that you must take to upgrade to the new Oracle Database 10g release depends on the release number of your current database. It may not be possible to upgrade directly from your current release of Oracle Database to the latest release. Depending on your current release, you may need to upgrade through one or more intermediate releases to upgrade to the new Oracle Database 10g release.For example, if the current database is running release 8.1.6, then first upgrade to release 8.1.7 using the instructions in Oracle8i Migration for release 8.1.7. The release 8.1.7 database can then be upgraded to the new Oracle Database 10g release.

Oracle9i database can coexist with Oracle Database 10g Release 2 (10.2). However, if you want separate releases of the database to coexist, then you must install Oracle Database 10g with Oracle9i already installed. You should not install Oracle9i after installing Oracle10g.

1.2.4 Hardware Requirements for Oracle Clusterware and Oracle Real Application Clusters

Each node in a cluster requires the following hardware:

• External shared disks for storing the Oracle Clusterware (Oracle Cluster Registry and voting disk) files, and database files.

  Chapter 3 describes the storage disk configuration options that are available. Review these options before you decide which storage option to use in your RAC environment. However, note that when Database Configuration Assistant (DBCA) configures automatic disk backup, it uses a database recovery area that must be shared.

  Note:

  Oracle Clusterware software can be installed on Oracle Cluster File System 2 (OCFS2). However, Oracle Clusterware software cannot be installed on Oracle Cluster File System (OCFS). Oracle Clusterware software can be installed on network-attached storage (NAS).

  For OCFS2 certification status, refer to the Certify page on OracleMetaLink:

  http://metalink.oracle.com
  
  

• One private internet protocol (IP) address for each node to serve as the private interconnect. The following must be true for each private IP address:

  • It must be separate from the public network

  • It must be accessible on the same network interface on each node

  • It must have a unique address on each node

  The private interconnect is used for internode communication by both Oracle Clusterware and RAC. The private IP address must be available in each node's /etc/hosts file.

  During Oracle Clusterware installation, the information you enter as the private IP address determines which private interconnects are used by Oracle Clusterware for its own communication. They must all be available, and capable of responding to a ping command.

  Oracle recommends that you use a logical Internet Protocol (IP) address that is available across all private networks, and that you take advantage of any available operating system-based failover mechanism by configuring it according to your third-party vendor's instructions for using their product to support failover.

• One public IP address for each node, to be used as the Virtual IP address for client connections and for connection failover.

  During installation this public virtual IP address (VIP) is associated with the same interface name on every node that is part of your cluster. The IP addresses that you use for all of the nodes that are part of a cluster must be from the same subnet. If you have a domain name server (DNS), then register the host names for the VIP with the DNS. The VIP should not be in use at the time of the installation, because this is a VIP that Oracle Clusterware manages.

• One public fixed host name address for each node, typically assigned by the system administrator during operating system installation. If you have a domain name server (DNS), then you can register both the fixed IP and the VIP address with the DNS. If you do not have a DNS, then you must make sure that both public IP addresses are in the node /etc/hosts file (for all cluster nodes), and any client system's /etc/hosts file that requires access to the database.

1.2.5 Software Requirements for Oracle Clusterware and Oracle Real Application Clusters

Each node in a cluster requires a supported interconnect software protocol to support Cache Fusion, and to support Oracle Clusterware polling. Your interconnect must be certified by Oracle for your platform. You should also have a Web browser, both to enable Oracle Enterprise Manager, and to view online documentation.

For Oracle Database 10g requirements, Oracle Clusterware provides the same functions as third-party vendor clusterware. Using Oracle Clusterware also reduces installation and support complications. However, you may require third-party vendor clusterware if you use a non-ethernet interconnect, or if you have deployed clusterware-dependent applications on the same cluster where you deploy RAC.

1.4.1 Pre-Installation Overview for Oracle Database 10g Real Application Clusters

The pre-installation procedures in Part II explain how to verify user equivalence, perform network connectivity tests, how to set directory and file permissions, and other required pre-installation tasks. Complete all pre-installation tasks and verify that your system meets all pre-installation requirements before proceeding to the install phase.

1.4.2 Installation Overview for Oracle Database 10g Real Application Clusters

Oracle Database 10g Real Application Clusters installation is a two-phase installation. In phase one, use Oracle Universal Installer (OUI) to install Oracle Clusterware as described in Chapter 4, "Installing Oracle Clusterware". Note that the Oracle home in phase one is a home for the Oracle Clusterware software, which must be different from the Oracle home that you use in phase two for the installation of the Oracle database software with RAC components. The Oracle Clusterware installation starts the Oracle Clusterware processes in preparation for installing Oracle Database 10g with RAC, as described in Chapter 5, "Installing Oracle Database 10g with Oracle Real Application Clusters". Use OUI in this phase to install the RAC software.

You must install Oracle Clusterware and Oracle Database in separate home directories. You should create the listener in the Oracle database Oracle home. If you will use multiple Oracle Database homes with ASM, then you should install a separate Oracle Database home for ASM. To create a new, independent ASM home, you should run the database OUI and select the option to install ASM.

If OUI detects a previous release of Oracle Clusterware (previously known as Oracle Cluster Ready Services), then you are prompted to select either a rolling upgrade, or a full upgrade.

If OUI detects a previous release of the Oracle database, then OUI provides you with the option to start Database Upgrade Assistant (DBUA) to upgrade your database to Oracle Database 10g Release 2 (10.2). In addition, DBUA displays a Service Configuration page for configuring services in your RAC database.

After the installation completes, OUI starts the Oracle Database assistants, such as Database Configuration Assistant (DBCA), to configure your environment and create your RAC database. You can later use the DBCA Instance Management feature to add or modify services and instances as described in Chapter 6, "Creating Oracle Real Application Clusters Databases with Database Configuration Assistant".

1.4.3 Post-Installation Overview for Oracle Database 10g Real Application Clusters

After you create your database, download and install the most recent patch set for your Oracle Database 10g release, as described in Chapter 7, "Oracle Real Application Clusters Post-Installation Procedures". If you are using other Oracle products with your RAC database, then you must also configure them.

You must also perform several post-installation configuration tasks to use certain Oracle Database 10g features.

On the installation media, you can select additional Oracle Database 10g software that may improve performance or extend database capabilities. Examples: Oracle JAccelerator, Oracle interMedia, and Oracle Text.

1.6.1 Overview of Automatic Storage Management

Oracle recommends using Automatic Storage Management (ASM) or a cluster file system with Oracle Managed Files (OMF) for database storage. This section provides an overview of ASM.

Note that RAC installations using Oracle Database Standard Edition must use ASM for database file storage.

You can use ASM to simplify the administration of Oracle database files. Instead of having to manage potentially thousands of database files, using ASM, you need to manage only a small number of disk groups. A disk group is a set of disk devices that ASM manages as a single logical unit. You can define a particular disk group as the default disk group for a database, and Oracle Database will automatically allocate storage for, create, or delete, the files associated with the appropriate database object. When administering the database, you need to refer to database objects only by name, rather than by file name.

When using ASM with a single Oracle home for database instances on a node, the ASM instance can run from that same home. If you are using ASM with Oracle Database instances from multiple database homes on the same node, then Oracle recommends that you run the ASM instance from an Oracle home that is distinct from the database homes. In addition, the ASM home should be installed on every cluster node. Following this recommendation prevents the accidental removal of ASM instances that are in use by databases from other homes during the de-installation of a database's Oracle home.

Benefits of Automatic Storage Management

ASM provides many of the same benefits as storage technologies such as a redundant array of independent disks (RAID) or a logical volume manager (LVM). Like these technologies, ASM lets you create a single disk group from a collection of individual disk devices. It balances input and output (I/O) loads to the disk group across all of the devices in the disk group. It also implements striping and mirroring to improve I/O performance and data reliability.

However, unlike RAID or LVMs, ASM implements striping and mirroring at the file level. This implementation lets you specify different storage attributes for individual files in the same disk group.

Disk Groups and Failure Groups

A disk group can contain between 1 to 10000 disk devices. Each disk device can be an individual physical disk, a multiple disk device such as a RAID storage array or logical volume, or even a partition on a physical disk. However, in most cases, disk groups consist of one or more individual physical disks. To enable ASM to balance I/O and storage appropriately within the disk group, all devices in the disk group should have similar, if not identical, storage capacity and performance.

When you add a device to a disk group, you can specify a failure group for that device. Failure groups define ASM disks that share a common potential failure mechanism. An example of a failure group is a set of SCSI disks sharing the same SCSI controller. Failure groups are used to determine which ASM disks to use for storing redundant copies of data. For example, if two-way mirroring is specified for a file, then ASM automatically stores redundant copies of file extents in separate failure groups. Failure groups apply only to normal and high redundancy disk groups. You define the failure groups in a disk group when you create or alter the disk group.

Redundancy Levels

ASM provides three levels of mirroring, called redundancy levels, that you can specify when creating a disk group. The redundancy levels are:

• External redundancy

  In disk groups created with external redundancy, the contents of the disk group are not mirrored by ASM. Choose this redundancy level when:

  • The disk group contains devices, such as RAID devices, that provide their own data protection

  • Your use of the database does not require uninterrupted access to data. For example: a development environment where you have a suitable backup strategy

• Normal redundancy

  In disk groups created with normal redundancy, the contents of the disk group are two-way mirrored by default. However, you can choose to create certain files that are three-way mirrored, or that are not mirrored. To create a disk group with normal redundancy, you must specify at least two failure groups (a minimum of two devices).

  The effective disk space of a disk group that uses normal redundancy is half the total disk space of all of its devices.

• High redundancy

  In disk groups created with high redundancy, the contents of the disk group are three-way mirrored by default. To create a disk group with high redundancy, you must specify at least three failure groups (a minimum of three devices).

  The effective disk space of a disk group that uses high redundancy is one-third of the total disk space of all of its devices.

ASM and Installation Types

The type and number of disk groups that you can create when installing Oracle Database software depends on the type of database you choose to create during the installation, as follows:

• Preconfigured database

  If you choose to create the default preconfigured database that uses ASM, then OUI prompts you to specify one or more disk device names and redundancy. By default, OUI creates a disk group named DATA, with normal redundancy.

• Advanced database

  If you choose to create an advanced database that uses ASM, then you can create one or more disk groups. These disk groups can use one or more devices. For each disk group, you can specify the redundancy level that suits your requirements.

• Configure Automatic Storage Management

  If you choose to create an ASM instance only, then OUI prompts you to create Disk Group. If OUI finds a Grid Control service is found on the system, then OUI prompts you to indicate if the ASM instance shall be managed by Grid Control. The Management Service box lists the available Oracle Management Services.

1.6.2 Shared Storage for Database Recovery Area

When you configure a database recovery area in a RAC environment, the database recovery area must be on shared storage. When Database Configuration Assistant (DBCA) configures automatic disk backup, it uses a database recovery area that must be shared.

If the database files are stored on a cluster file system, then the recovery area can also be shared through the cluster file system.

If the database files are stored on an Automatic Storage Management (ASM) disk group, then the recovery area can also be shared through ASM.

1.8.1 Oracle Clusterware

OUI installs Oracle Clusterware on each node of the cluster. If third-party vendor clusterware is not present, then you must use OUI to enter the nodes on which you want Oracle Clusterware to be installed. The Oracle Clusterware home can either be shared by all nodes, or private to each node, depending on your responses when you run OUI. The home that you select for Oracle Clusterware must be different from the RAC-enabled Oracle home.

When third-party vendor clusterware is present, Oracle Clusterware may interact with the third-party vendor clusterware. For Oracle Database 10g on Linux and Windows, Oracle Clusterware coexists with but does not interact with previous Oracle clusterware releases. In using third-party vendor clusterware, note the following:

• Oracle Clusterware can integrate with third-party vendor clusterware for all operating systems except Linux and Windows.

1.8.2 The Installed Real Application Clusters Components

All instances in RAC environments share the control file, server parameter file, redo log files, and all data files. These files reside on a shared cluster file system or on shared disks. Either of these types of file configurations are accessed by all the cluster database instances. Each instance also has its own set of redo log files. During failures, shared access to redo log files enables surviving instances to perform recovery.

1.11.1 Cloning Oracle Clusterware Homes

Complete the following tasks to clone an Oracle Clusterware home on multiple nodes:

1. On the source node, install Oracle Clusterware software. All required root scripts must run successfully.

2. As root, create a tar file of the Oracle Clusterware home

3. On the target node, create an Oracle Clusterware home, and copy the Oracle Clusterware tar file from the source node to the target node Oracle Clusterware home.

4. As the oracle user, uncompress the tar file.

5. Run OUI in clone mode, as described in Oracle Universal Installer and OPatch User's Guide.

6. Run root scripts.

7. Repeat steps 1 through 6 on each node that you want to add to the cluster. On the last node that you install, run the tool oifcfg to configure the network interfaces.

1.11.2 Cloning RAC Homes

Complete the following tasks to Clone a RAC database image on multiple nodes:

1. On the source node, install a RAC database Oracle home. All required root scripts must run successfully. Do not create a database, and do not run any configuration tools.

2. As root, create a tar file of the RAC database Oracle home.

3. On the target node, create an Oracle home directory for the RAC database, and copy the RAC database tar from the source node to the target node Oracle home.

4. Create required Oracle users and groups, ensuring that you use the same names, user ID numbers and group ID numbers as those on the source node.

5. As oracle, uncompress the tar file.

6. Run OUI in clone mode, as described in Oracle Universal Installer and OPatch User's Guide.

7. Run root scripts.

8. Repeat steps 1 through 7 for on each node that you want to add to the cluster.

9. Run the configuration assistant NetCA on a local node of the cluster, and provide a list when prompted of all nodes that are part of the cluster. This procedure creates the listener.

10. Run the configuration assistant DBCA to create the database.

11. Follow post-cloning phase instructions as provided in Oracle Universal Installer and OPatch User's Guide.