11 Adding and Deleting Nodes and Instances on Windows-Based Systems

Cloning Oracle Clusterware and Oracle RAC Software in Grid Environments

The preferred method for extending your Oracle RAC environment is to use the Oracle cloning procedures in the Oracle Universal Installer and OPatch User's Guide. Only use the procedures in this chapter if you choose not to use Oracle cloning.

Quick-Start Node and Database Instance Addition and Deletion Procedures

This section explains node and instance addition and deletion using procedures that are presented in a quick-start format. For node addition, these procedures require that you install the required operating system patches and updates on the new nodes. Then configure the new nodes to be part of your cluster at the network level. Extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with Oracle RAC components to the new nodes. Finally, make the new nodes members of the existing Oracle RAC database. This section includes the following topics:

• Adding an Oracle Clusterware Home to a New Node

• Adding an Oracle Home with Oracle RAC to a New Node

• Deleting an Oracle Home with Oracle RAC from an Existing Node

• Deleting an Oracle Clusterware Home from an Existing Node

Detailed Node and Database Instance Addition and Deletion Procedures

This section provides detailed procedures for adding and deleting nodes to clusters. The details of these steps appear in the following sections:

• Overview of Node Addition Procedures

• Step 1: Connecting New Nodes to the Cluster

• Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer

• Step 3: Preparing Storage on New Nodes

• Step 4: Adding Nodes at the Oracle RAC Database Layer

• Step 5: Adding Database Instances to New Nodes

• Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster

• Overview of Node Deletion Procedures

• Step 1: Deleting Instances from Oracle Real Application Clusters Databases

• Step 2: Deleting Nodes from Oracle Real Application Clusters Databases

• Step 3: ASM Instance Clean-Up Procedures for Node Deletion

Overview of Node Addition Procedures

This section explains how to add nodes to clusters using detailed manual procedures. Do this by setting up the new nodes to be part of your cluster at the network level. Then extend the Oracle Clusterware home from an existing Oracle Clusterware home to the new nodes and then extend the Oracle database software with Oracle RAC components to the new nodes. Finally, make the new nodes members of the existing Oracle RAC database.

If the nodes that you are adding to your cluster do not have Oracle Clusterware or Oracle software installed on them, then complete the following five steps. The procedures in these steps assume that you already have an operative Windows-based Oracle RAC environment. The details of these steps appear in the following sections.

• Step 1: Connecting New Nodes to the Cluster

• Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer

• Step 3: Preparing Storage on New Nodes

• Step 4: Adding Nodes at the Oracle RAC Database Layer

• Step 5: Adding Database Instances to New Nodes

To add a node to your cluster when the node is already configured with Oracle Clusterware and Oracle software, follow the procedure described in "Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster".

Step 1: Connecting New Nodes to the Cluster

Complete the following procedures to connect the new nodes to the cluster and to prepare them to support your Oracle RAC database:

• Making Physical Connections

• Installing the Operating System

• Verifying the Installation with the Cluster Verification Utility

• Checking the Installation

Step 2: Extending Oracle Software to New Nodes at the Oracle Clusterware Layer

The following section describes how to add new nodes to Oracle Clusterware and to the Oracle database software layers using OUI. The OUI requires access to the private interconnect that you checked as part of the installation validation in Step 1.

1. On one of the existing nodes go to the CRS home\oui\bin directory and run the addnode.bat script to start OUI.

2. The OUI runs in the add node mode and the OUI Welcome page appears. Click Next and the Specify Cluster Nodes for Node Addition page appears.

3. The upper table on the Specify Cluster Nodes for Node Addition page shows the existing nodes, the private node names and the virtual IP (VIP) addresses that are associated with Oracle Clusterware. Use the lower table to enter the public, private node names and the virtual hostnames of the new nodes.

4. Click Next and OUI verifies connectivity on the existing nodes and on the new nodes. The verifications that OUI performs include determining whether:

   • The nodes are up

   • The nodes are accessible by way of the network

     Note:

     If any of the existing nodes are down, then you can proceed with the procedure. However, once the nodes are up, you must run the following command on each of those nodes:

     setup.exe -updateNodeList -local 
     "CLUSTER_NODES={available_node_list}"
     ORACLE_HOME=CRS_home
     

     This operation should be run from the CRS_home\oui\bin directory and the available_node_list value is a comma-delimited list of all of nodes currently in the cluster and CRS_home defines the Oracle Clusterware home directory.

   • The virtual hostnames are not already in use on the network

   • The user has write permission to create the Oracle Clusterware home on the new nodes

   • The user has write permission to the OUI inventory in the C:\Program Files\Oracle\Inventory directory

5. If OUI detects that the new nodes do not have an inventory location, then:

   • The OUI automatically updates the inventory location in the Registry key

   If any verifications fail, then OUI re-displays the Specify Cluster Nodes for Node Addition page with a Status column in both tables indicating errors. Correct the errors or deselect the nodes that have errors and proceed. However, you cannot deselect existing nodes; you must correct problems on nodes that are already part of your cluster before you can proceed with node addition. If all of the checks succeed, then OUI displays the Node Addition Summary page.

6. The Node Addition Summary page displays the following information showing the products that are installed in the Oracle Clusterware home that you are extending to the new nodes:

   • The source for the add node process, which in this case is the Oracle Clusterware home

   • The private node names that you entered for the new nodes

   • The new nodes that you entered

   • The required and available space on the new nodes

   • The installed products listing the products that are already installed on the existing the Oracle Clusterware home

   Click Next and OUI displays the Cluster Node Addition Progress page.

7. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the phase of the node addition process and the phase's status according to the following:

   This page shows the following three OUI phases:

   • Copy the Oracle Clusterware Home to New Nodes: Copies the Oracle Clusterware home to the new nodes unless the Oracle Clusterware home is on the Oracle Cluster File System.

   • Perform Oracle Home Setup: Updates the Registry entries for the new nodes, creates the services, and creates folder entries.

   • Save Cluster Inventory: Updates the node list associated with the Oracle Clusterware home and its inventory.

   The Cluster Node Addition Progress page's Status column displays In Progress while the phase is in progress, Suspended when the phase is pending execution, and Succeeded after the phase completes. On completion, click Exit to end the OUI session. After OUI displays the End of Node Addition page, click Exit to end the OUI session.

8. From CRS_home\install on node1, run the crssetup.add.bat script.

9. Use the following command to perform an integrated validation of the Oracle Clusterware setup on all of the configured nodes, both the pre-existing nodes and the nodes that you have added

   cluvfy stage -post crinst -n all [-verbose]
   

The CVU -post crinst stage check verifies the integrity of the Oracle Clusterware components. After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to Step 3 to prepare the storage for Oracle RAC on the new nodes.

You can optionally run addnode.bat in silent mode, replacing steps 1 through 6 as follows, where nodeI, nodeI+1, and so on are the new nodes that you are adding:

addnode.bat -silent "CLUSTER_NEW_NODES={nodeI,nodeI+1,…nodeI+n}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={node-privI,node-privI+1,…node-privI+n}" 
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={node-vipI,node-vipI+1,…,node-vipI+n}"

You can alternatively specify the variable=value entries in a response file and run addnode as follows:

addnode.bat -silent -responseFile filename

Command-line values always override response file values.

Run crssetup.add.bat on the local node, or on the node on which you were performing this procedure.

After you have completed the procedures in this section for adding nodes at the Oracle Clusterware layer, you have successfully extended the Oracle Clusterware home from your existing the Oracle Clusterware home to the new nodes. Proceed to "Step 3: Preparing Storage on New Nodes" to prepare storage for Oracle RAC on the new nodes.

Step 3: Preparing Storage on New Nodes

To extend an existing Oracle RAC database to your new nodes, configure the shared storage for the new instances to be added on new nodes so that the storage type is the same as the storage that is already used by the existing nodes' instances. Prepare the same type of storage on the new nodes as you are using on the other nodes in the Oracle RAC environment that you want to extend as follows:

• Automatic Storage Management (ASM)

  If you are using ASM, then make sure that the new nodes can access the ASM disks with the same permissions as the existing nodes.

• Oracle Cluster File System (OCFS)

  If you are using Oracle Cluster File Systems, then make sure that the new nodes can access the cluster file systems in the same way that the other nodes access them.

• Raw Device Storage

  If your cluster database uses raw devices, then prepare the new raw devices by following the procedures described in"Raw Device Storage Preparation for New Nodes".

If you are using Oracle Cluster File System, then run the following command to verify your cluster file system and obtain detailed output where node_list includes both the pre-existing nodes and the newly added nodes:

cluvfy comp ssa -n node_list -s storageID_list [-verbose]

cluvfy comp ssa -n node_list -s storageID_list -verbose

Step 4: Adding Nodes at the Oracle RAC Database Layer

To add nodes at the Oracle RAC database later, run OUI in add node mode to configure your new nodes. If you have multiple Oracle homes, then perform the following steps for each Oracle home that you want to include on the new nodes:

1. On an existing node from the Oracle_home\oui\bin run the addnode.bat script. This starts OUI in the add node mode and displays the OUI Welcome page. Click Next on the Welcome page and OUI displays the Specify Cluster Nodes for Node Addition page.

2. The Specify Cluster Nodes for Node Addition page has a table showing the existing nodes associated with the Oracle home from which you launched OUI. A node selection table appears on the bottom of this page showing the nodes that are available for addition. Select the nodes that you want to add and click Next.

3. The OUI verifies connectivity and performs availability checks on both the existing nodes and on the nodes that you want to add. Some of checks performed determine whether:

4. • The nodes are up

   • The nodes are accessible by way of the network

   • The user has write permission to create the Oracle home on the new nodes

   • The user has write permission to the OUI inventory in the C:\Program Files\Oracle\Inventory directory on the existing nodes and on the new nodes

5. If the new nodes do not have an inventory set up, then OUI automatically updates the Registry entries for the inventory location. If any of the other checks fail, then fix the problem and proceed or deselect the node that has the error and proceed. You cannot deselect existing nodes; you must correct any problems on the existing nodes before proceeding with node addition. If all of the checks succeed, then OUI displays the Node Addition Summary page.

6. The Node Addition Summary page has the following information about the products that are installed in the Oracle home that you are going to extend to the new nodes:

   • The source for the add node process, which in this case is the Oracle home

   • The existing nodes and new nodes

   • The new nodes that you selected

   • The required and available space on the new nodes

   • The installed products listing all of the products that are already installed in the existing Oracle home

   Click Finish and OUI displays the Cluster Node Addition Progress page.

7. The Cluster Node Addition Progress page shows the status of the cluster node addition process. The table on this page has two columns showing the phase of the node addition process and the phase's status. The Cluster Node Addition Progress shows the following three OUI phases:

   • Copy the Oracle Home To New Nodes: Copies the entire Oracle home to the new nodes unless the Oracle home is on a cluster file system

   • Performs Oracle Home Setup: Updates the Registry entries for the new nodes, creates the services, and creates folder entries

   • Save Cluster Inventory: Updates the node list associated with the Oracle home and its inventory

   The Cluster Node Addition Progress page's Status column displays Succeeded if the phase completes, In Progress if the phase is in progress, and Suspended when the phase is pending execution. After OUI displays the End of Node Addition page, click Exit to end the OUI session.

8. Add a Listener to the new node by running the Net Configuration Assistant (NetCA) from the new node and selecting only the new node on the Node Selection page.

After completing the procedures in this section, you have defined the new nodes at the cluster database layer. You can now add database instances to the new nodes as described in Step 5.

You can optionally run addnode.bat in silent mode, replacing steps 1 through 6, as follows where nodeI, nodeI+1, and so on are the new nodes that you are adding:

addnode.bat -silent "CLUSTER_NEW_NODES={nodeI,nodeI+1,…nodeI+n}" 

The variable=value entries can also be specified in a response file, filename, and addnode can be run as:

addnode.bat -silent -responseFile filename

Command-line values always override response file values. Proceed to the next section, "Step 5: Adding Database Instances to New Nodes".

Step 5: Adding Database Instances to New Nodes

You can use Enterprise Manager, or the Database Configuration Assistant (DBCA) in either interactive mode or in silent mode, to add database instances to new nodes as described under the following headings:

• Using Enterprise Manager to Add Database Instances to New Nodes

• Using DBCA in Interactive Mode to Add Database Instances to New Nodes

• Using DBCA in Silent Mode to Add Database Instances to New Nodes

dbca -silent -addInstance -nodeList node -gdbName gdbname [-instanceName instname] -sysDBAUserName sysdba -sysDBAPassword password

Adding Nodes that Already Have Clusterware and Oracle Software to a Cluster

To add nodes to a cluster when the new nodes already have clusterware and Oracle software installed on them, you must configure the new nodes with the Oracle software that is on the existing nodes of the cluster. To do this, you must run two versions of an OUI process: one for the clusterware and one for the database layer as described in the following procedures:

1. Add new nodes at the Oracle Clusterware layer by running OUI using the following command:

   CRS_home\oui\bin\addnode.bat -noCopy 
   

2. Add new nodes at the Oracle software layer by running OUI from the Oracle home as follows:

   Oracle_home\oui\bin\addnode.bat -noCopy 
   

In the -noCopy mode, OUI performs all add node operations except for the copying of software to the new nodes.

Overview of Node Deletion Procedures

This section explains how to delete nodes from clusters using detailed manual procedures. The details of these steps appear in the following sections:

• Step 1: Deleting Instances from Oracle Real Application Clusters Databases

• Step 2: Deleting Nodes from Oracle Real Application Clusters Databases

• Step 3: ASM Instance Clean-Up Procedures for Node Deletion

Step 1: Deleting Instances from Oracle Real Application Clusters Databases

The procedures in this section explain how to use Enterprise Manager, or DBCA in interactive or silent mode, to delete an instance from an Oracle RAC database.

• Using Enterprise Manager to Delete Database Instances from Existing Nodes

• Using DBCA in Interactive Mode to Delete Database Instances from Existing Nodes

• Using DBCA in Silent Mode to Delete Instance from Existing Nodes

dbca -silent -deleteInstance [-nodeList node] -gdbName gdbname -instanceName instname -sysDBAUserName sysdba -sysDBAPassword password

Step 2: Deleting Nodes from Oracle Real Application Clusters Databases

Execute the following procedures to delete nodes from Oracle clusters on Windows-based systems.

Perform the following steps on a node other than the node that you want to delete:

1. Use the Database Configuration Assistant (DBCA) to delete the instance.

2. Use the Net Configuration Assistant (NETCA) to delete the Listener.

Perform the following steps on the node that you want to delete:

1. If the node that you are deleting has an ASM instance, then delete the ASM instance using the procedures in the following section, "Step 3: ASM Instance Clean-Up Procedures for Node Deletion".

2. On the that you are deleting, for example, node2, run the following command where location is the full path location of the Oracle home:

   setup.exe -updateNodeList ORACLE_HOME=location
   CLUSTER_ NODES="" -local
   

3. Depending on whether you have a shared or non-shared Oracle home, complete one of the following two procedures:

   • For a non-shared home, run OUI from the Oracle home and deinstall this home. You can run the setup.exe command from Oracle_home\oui\bin to start OUI so that you can deinstall the Oracle home.

   • For a shared home:

     • Run the following command from the Oracle_home\oui\bin directory:

       setup.exe –detachHome ORACLE_HOME=Oracle_home
       

     • Delete the services associated with this Oracle home.

     • Manually update the PATH environment variable by removing the entries associated with the Oracle home that you are removing.

     • Manually delete any Start menu folder items associated with the deleted Oracle home.

4. On node1, or in the case of multiple node installations, on any node other than the node that you are deleting, run the following command from Oracle_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle RAC installation:

   setup.exe –updateNodeList ORACLE_HOME=Oracle_home
   "CLUSTER_NODES={node_list}" 
   

5. From CRS_home\bin on any existing node, run the following command where remote_port is the number of the remote port:

   racgons remove_config node2:remote_port
   

   You can determine the remote port by viewing the contents of the file, CRS_ home\opmn\conf\ons.config.

6. Run SRVCTL to stop and remove the node applications from node2. From CRS_ home\bin run the following commands:

   srvctl stop nodeapps –n node2
   srvctl remove nodeapps –n node2
   

7. On node1, or on any node that you are not deleting, run the following command from CRS_home\bin where node_name is the node to be deleted and node_number is the node's number as obtained from the output of the olsnodes -n command:

   crssetup del –nn node_name,node_number
   

8. On the node or nodes that you are deleting, node2 in this example, run the following command from CRS_home\oui\bin:

   setup.exe –updateNodeList ORACLE_HOME=CRS_home 
   "CLUSTER_NODES={node2}" CRS=TRUE –local
   

9. On node2, using OUI setup.exe from CRS_home\oui\bin:

   • If you have a non-shared home, then deinstall the Oracle Clusterware installation by running the setup.exe script from CRS_home\oui\bin.

   • If you have a shared home, then do not perform a deinstallation. Instead, perform the following steps on the node that you want to delete:

     • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

       setup.exe –detachHome ORACLE_HOME=CRS_home
       

     • On node2, stop any services that are associated with this Oracle Clusterware home. The associated service s are namely: OracleCRSService, OracleEVMService, OracleCSService, and OracleClusterVolumeService.

     • Remove the Oracle services.

     • Manually delete all of the Start menu items associated with this Oracle Clusterware home.

     • Manually update the PATH environment variable to remove entries associated with this Clusterware home.

     • Then remove the following files under %SystemRoot%\System32\drivers: ocfs.sys, orafencedrv.sys, and orafenceservice.sys.

10. On node1, or in the case of a multiple node installation, on any node other than the one to be deleted, run the following command from CRS_home\oui\bin where node_list is a comma-delimited list of nodes that are to remain part of the Oracle RAC database:

    setup.exe –updateNodeList ORACLE_HOME=CRS_home
    "CLUSTER_NODES={node_list}" CRS=TRUE
    

As mentioned at the beginning of this procedure, you can optionally delete nodes from Oracle Real Application Clusters databases in silent mode by completing the following steps:

1. Perform steps 1 and 2 from the previous procedure.

2. On the node to be deleted, node2 in this case:

   • If you have a non-shared home, then deinstall the Oracle home as follows:

     setup.exe –silent  -deinstall "REMOVE_HOMES={Oracle_home}"
     

   • If the home is a shared home, then do not perform a deinstall. Instead, perform the following steps on the node to be deleted:

     • Perform a detach home on node2 by running the following command from Oracle_home\oui\bin on node2:

       setup.exe –detachHome –silent ORACLE_HOME=Oracle_home
       

     • On node 2, stop and delete any services that are associated with this Oracle home. In addition, delete any Registry entries and path entries that are associated with this Oracle home. Also, delete all of the start menu items associated with this Oracle home.

3. Perform steps 4 through 8 from the previous procedure.

4. On node2, using OUI setup.exe from CRS_home\oui\bin:

   • If you have a non-shared home, then deinstall the Oracle Clusterware home as follows:

     setup.exe –silent  -deinstall "REMOVE_HOMES={CRS_home}"
     

   • If you have a shared home, then do not perform a deinstallation. Instead, perform the following steps on the node that you want to delete:

     • Run the following command from CRS_home\oui\bin to detach the Oracle Clusterware home on node2:

       setup.exe –detachHome –silent ORACLE_HOME=CRS_home
       

     • On node2, stop any services that are associated with this Oracle Clusterware home. The associated service s are namely: OracleCRSService, OracleEVMService, OracleCSService, and OracleClusterVolumeService.

     • Remove the Oracle services.

     • Manually delete all of the Start menu items associated with this Oracle Clusterware home.

     • Manually update the PATH environment variable to remove entries associated with this Clusterware home.

     • Then remove the following files under %SystemRoot%\System32\drivers: ocfs.sys, orafencedrv.sys, and orafenceservice.sys.

5. Perform step 10 from the previous procedure.

Step 3: ASM Instance Clean-Up Procedures for Node Deletion

If you are using ASM, then the delete node procedure requires the following additional steps to remove the ASM instances:

1. If this is the Oracle home from which the node-specific Listener named LISTENER_nodename runs, then use NETCA to remove this Listener. If necessary, re-create this Listener in another home.

   See Also:

   Oracle Database Net Services Administrator's Guide for more information about NETCA

2. If this is the Oracle home from which the ASM instance runs, then remove the ASM configuration by running the following command for all nodes on which this Oracle home exists:

   srvctl stop asm -n node
   

   Then run the following command for the nodes that you are removing:

   srvctl remove asm -n node
   

3. Run the following command on each node that you are going to delete that has an ASM instance:

   oradim -delete -asmsid +ASMnode_number
   

4. If you are not using a cluster file system for your ASM Oracle home, then run the following commands on the deleted node:

5. rd /s /q ORACLE_BASE\admin\+ASM
   del Oracle_home\database\*ASM*