8 Using Oracle Clusterware to Manage Active Standby Pairs

Required privileges

See "ttCWAdmin" in Oracle TimesTen In-Memory Database Reference for information about the privileges required to execute ttCWAdmin commands.

Hardware and software requirements

TimesTen does not support Clusterware on Windows platforms.

TimesTen supports Oracle Clusterware releases 11.2.0.2 and 11.2.0.3 with TimesTen active standby pair replication. See Oracle Clusterware Administration and Deployment Guide for network and storage requirements and information about Oracle Clusterware configuration files.

Oracle Clusterware and TimesTen should be installed in the same location on all nodes. The TimesTen instance administrator must belong to the same UNIX primary group as the Oracle Clusterware installation owner.

All hosts should use Network Time Protocol (NTP) or a similar system so that clocks on the hosts remain within 250 milliseconds of each other. When adjusting the system clocks on any nodes to be synchronized with each other, do not set any clock backward in time.

Restricted commands and SQL statements

When you use Oracle Clusterware with TimesTen, the active standby pair replication scheme is created on the active master with the ttCWAdmin -create command and dropped with the ttCWAdmin -drop command. In between the ttCWAdmin -create and ttCWAdmin -drop commands, you cannot execute any of the following commands or SQL statements. However, you are able to perform these commands or SQL statements when you use the ttCWAdmin -beginAlterSchema and reinstated with the the ttCWAdmin -endAlterSchema command, as described in "Facilitating schema change for Oracle Clusterware".

• Creating, altering, or dropping the active standby pair with the CREATE ACTIVE STANDBY PAIR, ALTER ACTIVE STANDBY PAIR, and DROP ACTIVE STANDBY PAIR SQL statements

• Starting or stopping the replication agent with either the -repStart and -repStop options of the ttAdmin utility or the ttRepStart or ttRepStop built-in procedures.

• Starting or stopping the cache agent after the active standby pair has been created with either the -cacheStart and -cacheStop options of the ttAdmin utility or the ttCacheStart and ttCacheStop built-in procedures.

• Duplicating the database with the -duplicate option of the ttRepAdmin utility.

• Executing any built-in procedures for managing a cache grid when the active standby pair in a cluster is a member of a grid.

In addition, do not call ttDaemonAdmin -stop before calling ttCWAdmin -shutdown.

The TimesTen integration with Oracle Clusterware accomplishes these operations with the ttCWAdmin utility and the attributes specified in the cluster.oracle.ini file.

For more information about the built-ins and utilities, see Oracle TimesTen In-Memory Database Reference. For more information about the SQL statements, see Oracle TimesTen In-Memory Database SQL Reference.

Configuring basic availability

This example shows an active standby pair with no subscribers. The host for the active database is the first MasterHost defined (host1) and the standby database is the second MasterHost in the list (host2). Each host in the list is delimited by commas. You can include spaces for readability, if desired.

[basicDSN]
MasterHosts=host1,host2

The following is an example of a cluster.oracle.ini file for an active standby pair with one subscriber on host3:

[basicSubscriberDSN]
MasterHosts=host1,host2
SubscriberHosts=host3

Configuring advanced availability

Advanced availability involves configuring spare master or subscriber hosts that are idle until needed to replace master or subscriber hosts (used in the active standby pair replication scheme) that either shut down unexpectedly or experience an unrecoverable error.

As mentioned in "Configuring basic availability", the MasterHosts attribute in the cluster.oracle.ini file configures the hosts that are used as the master nodes. For an active standby pair replication scheme, you only need two master hosts. In the event of a failure, the host that did not fail becomes the active database (if not already the active database) and the failed host is recovered and becomes the standby master. However, if the failed host cannot be recovered and if you specified more than two hosts as master hosts in the cluster.oracle.ini file, then the next master host in the list can be instantiated to take the place of an unrecoverable master host.

For example, the following shows a configuration of several master hosts. The first two master hosts (host1 and host2) are used for the active and standby masters; the latter two master hosts (host3 and host4) can be used to take the place of either host1 or host2 if either encounter an unrecoverable failure.

MasterHosts=host1,host2,host3,host4

When you configure more than two multiple hosts, you should also configure two virtual IP (VIP) addresses used only by Oracle Clusterware resources that manage TimesTen resources. With these VIP addresses, TimesTen internal processes (those that manage replication) are isolated from any master host changes that may occur because of an unrecoverable host error.

These VIP addresses must be different from any other VIP addresses defined for Oracle Clusterware use or any VIP addresses that are to be used by user applications. Furthermore, if an application does use these VIP addresses, then the application may encounter errors when a master host fails (either recoverable or unrecoverable). These VIP addresses cannot be used by a user application as a method for client failover or as a method to isolate themselves if an active database and standby master switch.

Specify two VIP addresses in the MasterVIP parameter, one for each master host in the active standby pair replication scheme. The VIP addresses specified for the TimesTen cluster must be different from any VIP addresses already defined and used by Oracle Clusterware. In particular, the VIP addresses that are created during the Oracle Clusterware install cannot be used with TimesTen.

MasterVIP=192.168.1.1, 192.168.1.2

The following parameters are also associated with advanced availability in the cluster.oracle.ini file:

• SubscriberHosts, similar to MasterHosts lists the host names that can contain subscriber databases.

• SubscriberVIP, similar to MasterVIP, provides VIP addresses that can be used by TimesTen internally to manage a subscriber node.

• VIPInterface is the name of the public network adaptor.

• VIPNetMask defines the netmask of the virtual IP addresses.

In the following example, the hosts for the active database and the standby database are host1 and host2. The hosts available for instantiation in case of an unrecoverable error are host3 and host4. There are no subscriber nodes. VIPInterface is the name of the public network adaptor. VIPNetMask defines the netmask of the virtual IP addresses.

[advancedDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

The following example configures a single subscriber on host4. There is one extra host defined in SubscriberHosts that can be used for failover of the master databases and one extra node that can be used for failover of the subscriber database. MasterVIP and SubscriberVIP specify the virtual IP addresses defined for the master and subscriber hosts.

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4,host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

Ensure that the extra master nodes:

• Have TimesTen installed

• Have the direct-linked application installed if this is part of the configuration. See "Implementing application failover".

Including cache groups in the active standby pair

If the active standby pair replicates one or more AWT or read-only cache groups, set the CacheConnect attribute to y.

This example specifies an active standby pair with one subscriber in an advanced availability configuration. The active standby pair replicates one or more cache groups.

[advancedCacheDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0
CacheConnect=y

Including the active standby pair in a cache grid

If the active standby pair is a member of a cache grid, assign port numbers for the active and standby databases by setting the GridPort attribute.

This example specifies an active standby pair with no subscribers in an advanced availability configuration. The active standby pair is a member of a cache grid.

[advancedGridDSN]
MasterHosts=host1,host2,host3
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
CacheConnect=y
GridPort=16101, 16102

For more information about using Oracle Clusterware with a cache grid, see "Using Oracle Clusterware with a TimesTen cache grid".

Implementing application failover

TimesTen integration with Oracle Clusterware can facilitate the failover of a TimesTen application that is linked to any of the databases in the active standby pair. TimesTen can manage both direct-linked and client/server applications that are on the same host as Oracle Clusterware and TimesTen.

The required attributes in the cluster.oracle.ini file for failing over a TimesTen application are as follows:

• AppName - Name of the application to be managed by Oracle Clusterware

• AppStartCmd - Command line for starting the application

• AppStopCmd - Command line for stopping the application

• AppCheckCmd - Command line for executing an application that checks the status of the application specified by AppName

• AppType - Determines the database to which the application is linked. The possible values are Active, Standby, DualMaster, Subscriber (all) and Subscriber[index].

There are also several optional attributes that you can configure, such as AppFailureThreshold, DatabaseFailoverDelay, and AppScriptTimeout. All optional attributes (and their default values) are listed and described in Table A-3, "Optional attributes".

The TimesTen application monitor process uses the user-supplied script or program specified by AppCheckCmd to monitor the application. The script that checks the status of the application must be written to return 0 for success and a nonzero number for failure. When Oracle Clusterware detects a nonzero value, it takes action to recover the failed application.

This example shows advanced availability configured for an active standby pair with with no subscribers. The reader application is an application that queries the data in the standby database. AppStartCmd, AppStopCmd and AppCheckCmd can include arguments such as start, stop and check commands.

[appDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh start
AppStopCmd=/mycluster/reader/app_stop.sh stop
AppCheckCmd=/mycluster/reader/app_check.sh check

You can configure failover for more than one application. Use AppName to name the application and provide values for AppType, AppStartCmd, AppStopCmd and AppCheckCmd immediately following the AppName attribute. You can include blank lines for readability. For example:

[app2DSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0

AppName=reader
AppType=Standby
AppStartCmd=/mycluster/reader/app_start.sh
AppStopCmd=/mycluster/reader/app_stop.sh
AppCheckCmd=/mycluster/reader/app_check.sh

AppName=update
AppType=Active
AppStartCmd=/mycluster/update/app2_start.sh
AppStopCmd=/mycluster/update/app2_stop.sh
AppCheckCmd=/mycluster/update/app2_check.sh

If you set AppType to DualMaster, the application starts on both the active and the standby hosts. The failure of the application on the active host causes the active database and all other applications on the host to fail over to the standby host. You can configure the failure interval, the number of restart attempts, and the uptime threshold by setting the AppFailureInterval, AppRestartAttempts and AppUptimeThreshold attributes. These attributes have default values. For example:

[appDualDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AppName=update
AppType=DualMaster
AppStartCmd=/mycluster/update/app2_start.sh
AppStopCmd=/mycluster/update/app2_stop.sh
AppCheckCmd=/mycluster/update/app2_check.sh
AppRestartAttempts=5
AppUptimeThreshold=300
AppFailureInterval=30

Recovering from permanent failure of both master nodes

If both master nodes fail and then come back up, Oracle Clusterware can automatically recover the master databases. Automatic recovery of temporary dual failure requires the following:

• RETURN TWOSAFE is not specified for the active standby pair.

• AutoRecover is set to y.

• RepBackupDir specifies a directory on shared storage.

• RepBackupPeriod is set to a value greater than 0.

If both master nodes fail permanently, Oracle Clusterware can automatically recover the master databases to two new nodes if the following is true:

• Advanced availability is configured (virtual IP addresses and at least four hosts).

• The active standby pair does not replicate cache groups.

• A cache grid is not configured.

• RETURN TWOSAFE is not specified.

• AutoRecover is set to y.

• RepBackupDir specifies a directory on shared storage.

• RepBackupPeriod must be set to a value greater than 0.

TimesTen first performs a full backup of the active database and then performs incremental backups. You can specify the optional attribute RepFullBackupCycle to manage when TimesTen performs subsequent full backup. By default, TimesTen performs a full backup after every five incremental backups.

If RepBackupDir and RepBackupPeriod are configured for backups, TimesTen performs backups for any master database that becomes active. It does not delete backups that were performed for a database that used to be active and has become the standby unless the database becomes active again. Ensure that the shared storage has enough space for two complete database backups. The ttCWAdmin -restore command automatically chooses the correct backup files.

Incremental backups increase the amount of log records in the transaction log files. Ensure that the values of RepBackupPeriod and RepFullBackupCycle are small enough to prevent a large amount of log records in the transaction log file.

This example shows attribute settings for automatic recovery.

[autorecoveryDSN]
MasterHosts=host1,host2,host3,host4
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
AutoRecover=y
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

If you have cache groups in the active standby pair or prefer to recover manually from failure of both master hosts, ensure that AutoRecover is set to n (the default). Manual recovery requires the following:

• RepBackupDir specifies a directory on shared storage

• RepBackupPeriod must be set to a value greater than 0

This example shows attribute settings for manual recovery. The default value for AutoRecover is n, so it is not included in the file.

[manrecoveryDSN]
MasterHosts=host1,host2,host3
MasterVIP=192.168.1.1, 192.168.1.2
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

Using the RepDDL attribute

The RepDDL attribute represents the SQL statement that creates the active standby pair. The RepDDL attribute is optional. You can use it to exclude tables, cache groups and sequences from the active standby pair.

If you include RepDDL in the cluster.oracle.ini file, do not specify ReturnServiceAttribute, MasterStoreAttribute or SubscriberStoreAttribute in the cluster.oracle.ini file. Include those replication settings in the RepDDL attribute.

When you specify a value for RepDDL, use the <DSN> macro for the database file name prefix. Use the <MASTERHOST[1]> and <MASTERHOST[2]> macros to specify the master host names. TimesTen substitutes the correct values from the MasterHosts or MasterVIP attributes, depending on whether your configuration uses virtual IP addresses. Similarly, use the <SUBSCRIBERHOST[n]> macro to specify subscriber host names, where n is a number from 1 to the total number of SubscriberHosts attribute values or 1 to the total number of SubscriberVIP attribute values if virtual IP addresses are used.

Use the RepDDL attribute to exclude tables, cache groups, and sequences from the active standby pair:

[excludeDSN]
MasterHosts=host1,host2,host3,host4
SubscriberHosts=host5,host6
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>
SUBSCRIBER <DSN> ON <SUBSCRIBERHOST[1]>\
EXCLUDE TABLE pat.salaries, \
EXCLUDE CACHE GROUP terry.salupdate, \
EXCLUDE SEQUENCE ttuser.empcount

The replication agent transmitter obtains route information as follows, in order of priority:

1. From the ROUTE clause in the RepDDL setting, if a ROUTE clause is specified. Do not specify a ROUTE clause if you are configuring advanced availability.

2. From Oracle Clusterware, which provides the private host names and public host names of the local and remote hosts as well as the remote daemon port number. The private host name is preferred over the public host name. If the replication agent transmitter cannot connect to the IPC socket, it attempts to connect to the remote daemon using information that Oracle Clusterware maintains about the replication scheme.

3. From the active and standby hosts. If they fail, then the replication agent chooses the connection method based on host name.

This is an example of specifying the ROUTE clause in RepDDL:

[routeDSN]
MasterHosts=host1,host2,host3,host4
RepDDL=CREATE ACTIVE STANDBY PAIR \
<DSN> ON <MASTERHOST[1]>, <DSN> ON <MASTERHOST[2]>\
ROUTE MASTER <DSN> ON <MASTERHOST[1]>  SUBSCRIBER <DSN> ON <MASTERHOST[2]>\
MASTERIP "192.168.1.2" PRIORITY 1\
SUBSCRIBERIP "192.168.1.3" PRIORITY 1\ 
MASTERIP "10.0.0.1" PRIORITY 2\
SUBSCRIBERIP "10.0.0.2" PRIORITY 2\
MASTERIP "140.87.11.203" PRIORITY 3\
SUBSCRIBERIP "140.87.11.204" PRIORITY 3\
ROUTE MASTER <DSN> ON <MASTERHOST[2]>  SUBSCRIBER <DSN> ON <MASTERHOST[1]>\
MASTERIP "192.168.1.3" PRIORITY 1\
SUBSCRIBERIP "192.168.1.2" PRIORITY 1\ 
MASTERIP "10.0.0.2" PRIORITY 2\
SUBSCRIBERIP "10.0.0.1" PRIORITY 2\
MASTERIP "140.87.11.204" PRIORITY 3\
SUBSCRIBERIP "140.87.11.203" PRIORITY 3\

Install Oracle Clusterware

Install Oracle Clusterware. By default, the installation occurs on all hosts concurrently. See Oracle Clusterware installation documentation for your platform.

Oracle Clusterware starts automatically after successful installation.

Install TimesTen on each host

Install TimesTen in the same location on each host in the cluster, including extra hosts. The instance name must be the same on each host. The user name of the instance administrator must be the same on all hosts. The TimesTen instance administrator must belong to the same UNIX primary group as the Oracle Clusterware installation owner.

The installer prompts you for values for the following, each of which are stored in the ttcrsagent.options file:

• The TCP/IP port number associated with the TimesTen cluster agent. The port number must be the same on all nodes of the cluster. If you do not provide a port number, TimesTen uses the default TimesTen port.

• The Oracle Clusterware location. The location must be the same on each host.

• The hosts included in the cluster, including spare hosts, with host names separated by commas. This list must be the same on each host.

The ttCWAdmin –init and ttCWAdmin –shutdown commands use the ttcrsagent.options file to initiate and shut down the TimesTen cluster. The ttcrsagent.options file is located in the TimesTen daemon home directory.

Register the TimesTen cluster information

TimesTen cluster information is stored in the Oracle Cluster Registry (OCR). As the root user, enter this command:

ttCWAdmin -ocrConfig

As long as Oracle Clusterware and TimesTen are installed on the hosts, this step never needs to be repeated.

Start the TimesTen cluster agent

Start a TimesTen cluster agent and TimesTen cluster daemon monitor on all hosts in the cluster by executing the ttCWAdmin -init command. You can execute this command on any host in the cluster that is defined in the ttcrsagent.options file.

For example:

ttCWAdmin -init

The ttCWAdmin -init command performs the following:

• Reads the ttcrsagent.options file and launches the TimesTen main daemon on each of the hosts defined in this file.

• Starts and registers the TimesTen cluster agent (ttCRSAgent) and the TimesTen cluster daemon monitor (ttCRSDaemon) on the all hosts in the cluster. There is one TimesTen cluster agent and one TimesTen cluster daemon monitor for the TimesTen installation on each host. When the TimesTen cluster agent has started, Oracle Clusterware begins monitoring the TimesTen daemon on each host and restarts a TimesTen daemon if it fails.

To start and register the TimesTen cluster agent (ttCRSAgent) and the TimesTen cluster daemon monitor (ttCRSDaemon) on specific hosts in the cluster, use the -hosts command to specify the desired hosts in the cluster to start.

ttCWAdmin -init -hosts "host1, host2"

Create and populate a TimesTen database on one host

Create a database on the host where you intend the active database to reside. The DSN must be the same as the database file name.

Create schema objects (such as tables, AWT cache groups, and read-only cache groups) and populate with data as appropriate. However, before you create cache groups, you must first decide when to load the cache groups.

• For best performance, load the cache group tables from the Oracle database tables before the ttCWAdmin -create command. There is less performance overhead when cache groups are loaded with initial data before the duplicate is performed on the active master to create the standby master (and any subscribers).

  For this option, perform the following:

  1. Start the cache agent as follows:

     Command> call ttCacheStart;
     

     Note:

     Since this is before the ttCWAdmin -start command, you can start the cache agent at this time. The ttCWAdmin -start command notes that the cache agent is already started and continues.

  2. Use the LOAD CACHE GROUP statement to load the cache group tables from the Oracle database tables.

  3. If using autorefresh cache groups, set the autorefresh state to pause with the ALTER CACHE GROUP SET AUTOREFRESH STATE PAUSED statement. The autorefresh state will be set to ON as part of the ttCWAdmin -start command.

  The following example demonstrates how to create a read-only autorefresh cache group, load the data, and then set the autorefresh state to pause:

  Command> call ttCacheStart;
  Command> CREATE READONLY CACHE GROUP my_cg
    AUTOREFRESH MODE INCREMENTAL INTERVAL 60 SECONDS
    FROM t1 (c1 NUMBER(22) NOT NULL PRIMARY KEY, c2 DATE, c3 VARCHAR(30));
  
  Command> LOAD CACHE GROUP my_cg COMMIT EVERY 100 ROWS PARALLEL 4;
  Command> ALTER CACHE GROUP my_cg SET AUTOREFRESH STATE PAUSED;
  

• Alternatively, wait to load the cache group tables after the ttCWAdmin -start as described in "Load cache groups". The data will be replicated to the standby master and any subscribers.

Create system DSN files on other hosts

On all hosts that are to be included in the cluster, create the system DSN (sys.odbc.ini) files. The DataStore attribute and the Data Source Name must be the same as the entry name for the cluster.oracle.ini file. See "Configuring Oracle Clusterware management with the cluster.oracle.ini file" for information about the contents of the sys.odbc.ini files.

Create a cluster.oracle.ini file

Create a cluster.oracle.ini file as a text file. See "Configuring Oracle Clusterware management with the cluster.oracle.ini file" for details about its contents and acceptable locations for the file.

Create the Oracle Clusterware resources to manage virtual IP addresses

Advanced availability involves configuring spare master or subscriber hosts that are idle until needed to replace master or subscriber hosts (used in the active standby pair replication scheme) that either shut down unexpectedly or experience an unrecoverable error. This is an optional step that is only necessary if you decide to configure advanced availability.

If you are planning on providing additional master or subscriber hosts for advanced availability, then you need to configure virtual IP addresses (one for each master host and subscriber actively used in the active standby pair). See "Configuring advanced availability" for more details on how many virtual IP addresses should be created.

In this case, perform the following:

1. Designate (or create) new virtual IP addresses on the network that are to be used solely for managing multiple hosts in a TimesTen replication environment managed by Oracle Clusterware.

2. Configure these VIP addresses for use to manage multiple hosts for advanced availability in the cluster.oracle.ini file, as described in "Configuring advanced availability".

3. Create the Oracle Clusterware resources that manage these VIP addresses by executing the ttCWAdmin -createVIPs command as the root user on any host in the cluster.

   For example:

   ttCWAdmin -createVIPs -dsn myDSN
   

   The VIP address names created by this command start with network_ followed by the TimesTen instance name, TimesTen instance administrator, and the DSN. Whereas, the VIP addresses created for use by Oracle Clusterware are prefixed with ora.

   Note:

   The ttCWAdmin -createVIPs command must be executed before the ttCWAdmin -create command. If you decide that you want to use VIP addresses for advanced availability after you have executed the ttCWAdmin -create command, you must perform the following:

   1. Execute ttCWadmin -drop to drop the active standby pair replication scheme.

   2. Add VIP addresses into cluster.oracle.ini file.

   3. Execute ttCWadmin -createVIPs to create the resources to manage the VIP addresses.

   4. Execute ttCWAdmin -create to create the active standby pair replication scheme managed by Oracle Clusterware.

Once created, the only way to drop the Oracle Clusterware resources that manage the VIP addresses is to execute the ttCWAdmin -dropVIPs command. Before you can drop the virtual IP addresses, you must first execute the ttCWAdmin -drop command.

The following is an example of how to drop the virtual IP addresses:

ttCWAdmin -dropVIPs -dsn myDSN 

For an example of when to use the ttCWAdmin -dropVIPs command, see "Removing an active standby pair from a cluster".

Create an active standby pair replication scheme

Create an active standby pair replication scheme by executing the ttCWAdmin -create command on any host in the cluster.

For example:

ttCWAdmin -create -dsn myDSN

This command prompts for an encryption pass phrase that the user does not need again. The command also prompts for the user ID and password for a TimesTen user with the ADMIN privilege if it does not find this information in the sys.odbc.ini file. This TimesTen user is used to create the active standby pair.

If the CacheConnect Clusterware attribute is enabled, the command prompts for the user password for the Oracle database. The Oracle database password is used to set the autorefresh states for cache groups. See "CacheConnect" for more details on this attribute.

If you want to specify the path and name of a file to be used as the cluster.oracle.ini file, use the -ttclusterini command.

ttCWAdmin -create -dsn myDSN -ttclusterini path/to/cluster/mycluster.ini

To drop the active standby pair, use the ttCWAdmin -drop command, as follows:

ttCWAdmin -drop -dsn myDSN

For examples showing the sequence in which to use the ttCWAdmin -create and ttCWAdmin -drop commands, see "Managing active standby pairs in a cluster" and "Managing read-only subscribers in the active standby pair".

Start the active standby pair and the applications

Start the cluster with the active standby pair replication scheme by executing the ttCWAdmin -start command on any host. This starts the cache agent (if not already started) and replication agent on the active database, performs the duplicate to create the standby master (and any subscribers), and starts the cache agent and replication agent on the standby master (and any subscribers).

If you do not specify -noApp, the applications are also started. If you do specify -noApp, then you can start and stop the applications with the -startApps and -stopApps commands respectively.

For example:

ttCWAdmin -start -dsn myDSN

This command starts the following processes for the active standby pair:

• ttCRSMaster

• ttCRSActiveService

• ttCRSsubservice

• Monitor for application AppName

The following example starts the cache and replication agents, but does not start the applications because of the inclusion of the -noapp command:

ttCWAdmin -start -noapp -dsn myDSN

To start and stop applications, use the -startApps and -stopApps commands as shown below:

ttCWAdmin -startapps -dsn myDSN
...
ttCWAdmin -stopapps -dsn myDSN

To stop the cache agent and replication agent and disconnect the application from both databases, execute the ttCWAdmin -stop command.

ttCWAdmin -stop -dsn myDSN

For examples showing the sequence in which to use the ttCWAdmin -start and ttCWAdmin -stop commands, see "Managing active standby pairs in a cluster" and "Managing read-only subscribers in the active standby pair".

Load cache groups

If the active standby pair includes cache groups and you have not already loaded the cache group (as described in "Create and populate a TimesTen database on one host"), use the LOAD CACHE GROUP statement to load the cache group tables from the Oracle database tables.

For more information on when to load cache groups, see "Create and populate a TimesTen database on one host".

Include more than one active standby pair in a cluster

If you want to use Oracle Clusterware to manage more than one active standby pair in a cluster, include additional configuration in the cluster.oracle.ini file. Oracle Clusterware can only manage more than one active standby pair in a cluster if all TimesTen databases are a part of the same TimesTen instance on a single host.

For example, the following cluster.oracle.ini file contains configuration information for two active standby pair replication schemes on the same host:

[advancedSubscriberDSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
MasterVIP=192.168.1.1, 192.168.1.2
SubscriberVIP=192.168.1.3
VIPInterface=eth0
VIPNetMask=255.255.255.0

[advSub2DSN]
MasterHosts=host1,host2,host3
SubscriberHosts=host4, host5
MasterVIP=192.168.1.4, 192.168.1.5
SubscriberVIP=192.168.1.6
VIPInterface=eth0
VIPNetMask=255.255.255.0

Perform these tasks for additional replication schemes:

1. Create and populate the databases.

2. Create the virtual IP addresses. Use the ttCWAdmin -createVIPs command.

3. Create the active standby pair replication scheme. Use the ttCWAdmin -create command.

4. Start the active standby pair. Use the ttCWAdmin -start command.

Configure an Oracle database as a disaster recovery subscriber

You can create an active standby pair on the primary site with an Oracle database as a remote disaster recovery subscriber. See "Using a disaster recovery subscriber in an active standby pair". Oracle Clusterware manages the active standby pair, but does not manage the disaster recovery subscriber. The user must perform a switchover if the primary site fails.

To use Oracle Clusterware to manage an active standby pair that has a remote disaster recovery subscriber, perform these tasks:

1. Use the RepDDL or RemoteSubscriberHosts Clusterware attribute to provide information about the remote disaster recovery subscriber. For example:

   [advancedDRsubDSN]
   MasterHosts=host1,host2,host3
   SubscriberHosts=host4, host5
   RemoteSubscriberHosts=host6
   MasterVIP=192.168.1.1, 192.168.1.2
   SubscriberVIP=192.168.1.3
   VIPInterface=eth0
   VIPNetMask=255.255.255.0
   CacheConnect=y
   

2. Use ttCWAdmin -create to create the active standby pair replication scheme on the primary site. This does not create the disaster recovery subscriber.

3. Use ttCWAdmin -start to start the active standby pair replication scheme.

4. Load the cache groups that are replicated by the active standby pair.

5. Set up the disaster recovery subscriber using the procedure in "Rolling out a disaster recovery subscriber".

Configure a read-only subscriber that is not managed by Oracle Clusterware

You can include a read-only TimesTen subscriber database that is not managed by Oracle Clusterware. Perform these tasks:

1. Include the RemoteSubscriberHosts Clusterware attribute in the cluster.oracle.ini file. For example:

   [advancedROsubDSN]
   MasterHosts=host1,host2,host3
   RemoteSubscriberHosts=host6
   MasterVIP=192.168.1.1, 192.168.1.2
   SubscriberVIP=192.168.1.3
   VIPInterface=eth0
   VIPNetMask=255.255.255.0
   

2. Use ttCWAdmin -create to create the active standby pair replication scheme on the primary site.

3. Use ttCWAdmin -start to start the active standby pair replication scheme. This does not create the read-only subscriber.

4. Use the ttRepStateGet procedure to verify that the state of the standby database is STANDBY.

5. On the subscriber host, use ttRepAdmin -duplicate option to duplicate the standby database to the read-only subscriber. See "Duplicating a database".

6. Start the replication agent on the subscriber host.

To add a read-only subscriber to an existing configuration, see "Adding a read-only subscriber not managed by Oracle Clusterware".

To rebuild a read-only subscriber, see "Rebuilding a read-only subscriber not managed by Oracle Clusterware".

Creating and initializing a cluster of cache grid members

See "Install TimesTen on each host" for installation requirements. In addition, each grid member must have a DSN that is unique within the cache grid.

Perform the tasks described in "Creating and initializing a cluster" for each grid member. Include the GridPort Clusterware attribute in the cluster.oracle.ini file as described in "Including the active standby pair in a cache grid". Ensure that the specified port numbers are not in use.

The ttCWAdmin -start command automatically attaches a grid member to the cache grid attach. The ttCWAdmin -stop command automatically detaches a grid member from the cache grid.

Failure and recovery for active standby pair grid members

If both nodes of an active standby pair grid member fail, then the grid member fails. Oracle Clusterware evicts the failed grid member from the grid automatically. However, when a cache grid is configured, any further automatic recovery after a dual failure, whether temporary or permanent, is not possible. In this case, you can only recover manually. For details, see "Manual recovery of both nodes of an active standby pair grid member".

Making schema changes to active standby pairs in a grid

You can add, drop, or change a cache group while the active database is attached to the grid.

Use the ttCWAdmin -beginAlterSchema command to make these schema changes. This command stops the replication agents on both the active and standby master nodes, but allows the active database to remain attached to the grid. The ttCWAdmin -endAlterSchema command duplicates the changes to the standby database (if there are changes to any objects that are replicated), registers the altered replication scheme, and starts both replication agents.

To add or drop a replicated table into the active standby pair, see "Making DDL changes in an active standby pair".

How TimesTen performs recovery when Oracle Clusterware is configured

The TimesTen database monitor (ttCRSmaster process) performs recovery. It attempts to connect to the failed database without using the forceconnect option. If the connection fails with error 994 (Data store connection terminated), the database monitor tries to connect 10 times. If the connection fails with error 707 (Attempt to connect to a data store that has been manually unloaded from RAM), the database monitor changes the RAM policy and tries to connect again. If the database monitor cannot connect, it returns connection failure.

If the database monitor can connect to the database, then it performs these tasks:

• It queries the CHECKSUM column in the TTREP.REPLICATIONS replication table.

• If the value in the CHECKSUM column matches the checksum stored in the Oracle Cluster Registry, then the database monitor verifies the role of the database. If the role is 'ACTIVE', then recovery is complete.

  If the role is not 'ACTIVE', then the database monitor queries the replication Commit Ticket Number (CTN) in the local database and the CTN in the active database to find out whether there are transactions that have not been replicated. If all transactions have been replicated, then recovery is complete.

• If the checksum does not match or if some transactions have not been replicated, then the database monitor performs a duplicate operation from the remote database to re-create the local database.

If the database monitor fails to connect with the database because of error 8110 or 8111 (master catchup required or in progress), then it uses the forceconnect=1 option to connect and starts master catchup. Recovery is complete when master catchup has been completed. If master catchup fails with error 8112 (Operation not permitted), then the database monitor performs a duplicate operation from the remote database. For more information about master catchup, see "Automatic catch-up of a failed master database".

If the connection fails because of other errors, then the database monitor tries to perform a duplicate operation from the remote database.

The duplicate operation verifies that:

• The remote database is available.

• The replication agent is running.

• The remote database has the correct role. The role must be 'ACTIVE' when the duplicate operation is attempted for creation of a standby database. The role must be 'STANDBY' or 'ACTIVE' when the duplicate operation is attempted for creation of a read-only subscriber.

When the conditions for the duplicate operation are satisfied, the existing failed database is destroyed and the duplicate operation starts.

When an active database or its host fails

If there is a failure on the node where the active database resides, Oracle Clusterware automatically changes the state of the standby database to 'ACTIVE'. If application failover is configured, then the application begins updating the new active database.

Figure 8-2 shows that the state of the standby database has changed to 'ACTIVE' and that the application is updating the new active database.

Figure 8-2 Standby database becomes active

Description of "Figure 8-2 Standby database becomes active"

Oracle Clusterware tries to restart the database or host where the failure occurred. If it is successful, then that database becomes the standby database.

Figure 8-3 shows a cluster where the former active node becomes the standby node.

Figure 8-3 Standby database starts on former active host

Description of "Figure 8-3 Standby database starts on former active host"

If the failure of the former active node is permanent and advanced availability is configured, Oracle Clusterware starts a standby database on one of the extra nodes.

Figure 8-4 shows a cluster in which the standby database is started on one of the extra nodes.

Figure 8-4 Standby database starts on extra host

Description of "Figure 8-4 Standby database starts on extra host"

If you do not want to wait for these automatic actions to occur, see "Perform a forced switchover after failure of the active database or host".

When a standby database or its host fails

If there is a failure on the standby node, Oracle Clusterware first tries to restart the database or host. If it cannot restart the standby database on the same host and advanced availability is configured, Oracle Clusterware starts the standby database on an extra node.

Figure 8-5 shows a cluster in which the standby database is started on one of the extra nodes.

Figure 8-5 Standby database on new host

Description of "Figure 8-5 Standby database on new host"

When read-only subscribers or their hosts fail

If there is a failure on a subscriber node, Oracle Clusterware first tries to restart the database or host. If it cannot restart the database on the same host and advanced availability is configured, Oracle Clusterware starts the subscriber database on an extra node.

When failures occur on both master nodes

This section includes these topics:

• Automatic recovery when not attached to a grid

• Manual recovery of both nodes of an active standby pair grid member

• Manual recovery for advanced availability

• Manual recovery for basic availability

• Manual recovery to the same master nodes when databases are corrupt

• Manual recovery when RETURN TWOSAFE is configured

[basicTwosafeDSN]
MasterHosts=host1,host2
ReturnServiceAttribute=RETURN TWOSAFE
RepBackupDir=/shared_drive/dsbackup
RepBackupPeriod=3600

When more than two master hosts fail

Approach a failure of more than two master hosts as a more extreme case of dual host failure. Use these guidelines:

• Address the root cause of the failure if it is something like a power outage or network failure.

• Identify or obtain at least two healthy hosts for the active and standby databases.

• Update the MasterHosts and SubscriberHosts entries in the cluster.oracle.ini file.

• See "Manual recovery for advanced availability" and "Manual recovery for basic availability" for guidelines on subsequent actions to take.

Perform a forced switchover after failure of the active database or host

If you want to force a switchover to the standby database without waiting for automatic recovery to be performed by TimesTen and Oracle Clusterware, you can write an application that uses Oracle Clusterware commands.

Perform the following:

1. Use the crsctl stop resource command to stop the ttCRSmaster resource on the active database. This causes the role of the standby database to change to active.

2. Use the crsctl start resource command to restart the ttCRSmaster resource on the former active database. This causes the database to recover and become the standby database.

The following example demonstrates a forced switchover from the active database on host1 to the standby database on host2.

1. Find all TimesTen resources using the crsctl status resource command.

   % crsctl status resource | grep TT
     NAME=TT_Activeservice_tt1122_ttadmin_REP1
     NAME=TT_Agent_tt1122_ttadmin_HOST1
     NAME=TT_Agent_tt1122_ttadmin_HOST2
     NAME=TT_App_tt1122_ttadmin_REP1_updateemp
     NAME=TT_Daemon_tt1122_ttadmin_HOST1
     NAME=TT_Daemon_tt1122_ttadmin_HOST2
     NAME=TT_Master_tt1122_ttadmin_REP1_0
     NAME=TT_Master_tt1122_ttadmin_REP1_1
     NAME=TT_Subservice_tt1122_ttadmin_REP1
   

2. Find the host where the active database resides by retrieving the status of the ttCRSActiveService resource.

   % crsctl status resource TT_Activeservice_tt1122_ttadmin_REP1
     NAME=TT_Activeservice_tt1122_ttadmin_REP1
     TYPE=application
     TARGET=ONLINE
     STATE=ONLINE on host1
   

3. There are two ttCRSmaster resources listed in the initial status report. Discover which ttCRSmaster resource is on the same host as the active database.

   % crsctl status resource TT_Master_tt1122_ttadmin_REP1_0
     NAME=TT_Master_tt1122_ttadmin_REP1_0
     TYPE=application
     TARGET=ONLINE
     STATE=ONLINE on host1
   
   % crsctl status resource TT_Master_tt1122_ttadmin_REP1_1
     NAME=TT_Master_tt1122_ttadmin_REP1_1
     TYPE=application
     TARGET=ONLINE
     STATE=ONLINE on host2
   

4. Stop the ttCRSmaster resource on the host where the active database resides.

   % crsctl stop resource TT_Master_tt1122_ttadmin_REP1_0
     CRS-2673: Attempting to stop 'TT_Master_tt1122_ttadmin_REP1_0'
     on 'host1'
     CRS-2677: Stop of 'TT_Master_tt1122_ttadmin_REP1_0' on
    'host1' succeeded
   

5. Restart the ttCRSmaster resource on the former active database.

   % crsctl start resource TT_Master_tt1122_ttadmin_REP1_0
     CRS-2672: Attempting to start 'TT_Master_tt1122_ttadmin_REP1_0'
     on 'host1'
     CRS-2676: Start of 'TT_Master_tt1122_ttadmin_REP1_0' on
    'host1' succeeded
   

6. Confirm that the forced switchover succeeds by checking where the ttCRSActiveService and ttCRSsubservice resources are located.

   % crsctl status resource TT_Activeservice_tt1122_ttadmin_REP1
     NAME=TT_Activeservice_tt1122_ttadmin_REP1
     TYPE=application
     TARGET=ONLINE
     STATE=ONLINE on host2
    
   % crsctl status resource TT_Subservice_tt1122_ttadmin_REP1
     NAME=TT_Subservice_tt1122_ttadmin_REP1
     TYPE=application
     TARGET=ONLINE
     STATE=ONLINE on host1
   

See Oracle Clusterware Administration and Deployment Guide for more information about the crsctl start resource and crsctl stop resource commands.

Performing a rolling upgrade of Oracle Clusterware software

See Oracle Clusterware Administration and Deployment Guide.

Upgrading TimesTen

See "Performing an online TimesTen upgrade when using Oracle Clusterware" in Oracle TimesTen In-Memory Database Installation Guide.

Managing hosts in the cluster

The following sections describe how to add or remove hosts when using a cluster:

• Adding a host to the cluster

• Removing a host from the cluster

ttCWAdmin -addMasterHosts -hosts "host8,host9" -dsn advancedSubscriberDSN

ttCWAdmin -addSubscriberHosts -hosts "subhost1" -dsn advancedSubscriberDSN

ttCWAdmin -delMasterHosts "host8,host9" -dsn advancedSubscriberDSN

ttCWAdmin -delSubscriberHosts "subhost1" -dsn advancedSubscriberDSN

Managing active standby pairs in a cluster

The following sections describe how to add or remove an active standby pair to a cluster:

• Adding an active standby pair to a cluster

• Removing an active standby pair from a cluster

Managing read-only subscribers in the active standby pair

The following sections describe how to manage read-only subscribers in the active standby pair that is managed by Oracle Clusterware:

• Adding a read-only subscriber managed by Oracle Clusterware

• Removing a read-only subscriber managed by Oracle Clusterware

• Adding a read-only subscriber not managed by Oracle Clusterware

• Rebuilding a read-only subscriber not managed by Oracle Clusterware

Changing the schema

The "Making DDL changes in an active standby pair" and "Making other changes to an active standby pair" sections describe what DDL statements are and are not automatically replicated for an active standby pair. When using Oracle Clusterware to manage an active standby pair, you can execute the DDL statements as in a normal replication environment, except that Oracle Clusterware must control the starting and stopping of the replication agents.

Thus, when you are changing the schema, note the following:

• For any DDL statements mentioned in "Making DDL changes in an active standby pair" that are automatically replicated, no further action is required. The DDLReplicationLevel connection attribute controls what DDL statements are replicated.

• For any DDL statements that are not replicated and are executed on non-replicated objects (view, PL/SQL function, PL/SQL procedure, PL/SQL package, or PL/SQL package body), manually execute the DDL statement on each node in the replication scheme. You do not need to stop the replication agents for these objects.

• Any DDL statements on objects that are not replicated but are part of the replication scheme, which are mentioned in "Making other changes to an active standby pair", you must first stop the replication agent on each node in the replication scheme with the Oracle Clusterware -beginAlterSchema command. Then, execute the DDL statement on the active database in the replication scheme. Finally, restart all replication agents with the Oracle Clusterware -endAlterSchema command. Follow the instructions described in "Facilitating schema change for Oracle Clusterware".

• Any DDL statements to be executed on materialized views, you must first stop the replication agent on each node in the replication scheme with the Oracle Clusterware -beginAlterSchema command. Then, you must manually execute the DDL statement on each node in the replication scheme. Finally, restart all replication agents with the Oracle Clusterware -endAlterSchema command. Follow the instructions described in "Facilitating schema change for Oracle Clusterware".

Reversing the roles of the master databases

After a failover, the active and standby databases are on different hosts than they were before the failover. You can use the -switch option of the ttCWAdmin utility to restore the original configuration. Optionally, you can also use the -timeout option with the -switch option to set a timeout for the number of seconds to wait for the active and standby database switch to complete.

For example:

ttCWAdmin -switch -dsn basicDSN

Ensure that there are no open transactions before using the -switch option. If there are open transactions, the command fails.

Figure 8-6 shows the hosts for an active standby pair. The active database resides on host A, and the standby database resides on host B.

Figure 8-6 Hosts for an active standby pair

Description of "Figure 8-6 Hosts for an active standby pair"

The ttCWAdmin -switch command performs these tasks:

• Deactivates the TimesTen cluster agent (ttCRSAgent) on host A (the active node).

• Disables the database monitor (ttCRSmaster) on host A.

• Calls the ttRepSubscriberWait, ttRepStop and ttRepDeactivate built-in procedures on host A.

• Stops the active service (ttCRSActiveService) on host A and reports a failure event to the Oracle Clusterware CRSD process.

• Enables monitoring on host A and moves the active service to host B.

• Starts the replication agent on host A, stops the standby service (ttCRSsubservice) on host B and reports a failure event to the Oracle Clusterware CRSD process on host B.

• Starts the standby service (ttCRSsubservice) on host A.

Moving a database to a different host

When a cluster is configured for advanced availability, you can use the -relocate option of the ttCWAdmin utility to move a database from the local host to the next available spare host specified in the MasterHosts attribute in the cluster.oracle.ini file. If the database on the local host has the active role, the -relocate option first reverses the roles. Thus, the newly migrated active database becomes the standby and the standby becomes the active.

The -relocate option is useful for relocating a database if you decide to take the host offline. Ensure that there are no open transactions before you use the command.

If the -relocate option requires a role switch, then you can optionally use the -timeout option with the -relocate option to set a timeout for the number of seconds to wait for the role switch.

For example:

ttCWAdmin -relocate -dsn advancedDSN

Performing host or network maintenance

If you decide to upgrade the operating system or hardware for a host or perform network maintenance, shut down Oracle Clusterware and disable automatic startup. Execute these Oracle Clusterware commands as root or OS administrator:

# crsctl stop crs

# crsctl disable crs

Shut down TimesTen. See "Shutting down a TimesTen application" in Oracle TimesTen In-Memory Database Operations Guide.

Perform the host maintenance. Then enable automatic startup and start Oracle Clusterware:

# crsctl enable crs

# crsctl start crs

See Oracle Clusterware Administration and Deployment Guide for more information about these commands.

Performing maintenance on the entire cluster

When all of the hosts in the cluster need to be brought down, stop Oracle Clusterware on each host individually. Execute these Oracle Clusterware commands as root or OS administrator:

# crsctl stop crs

# crsctl disable crs

Shut down TimesTen. See "Shutting down a TimesTen application" in Oracle TimesTen In-Memory Database Operations Guide.

Perform the maintenance. Then enable automatic startup and start Oracle Clusterware:

# crsctl enable crs

# crsctl start crs

See Oracle Clusterware Administration and Deployment Guide for more information about these commands.

Changing user names or passwords

When you create the active standby pair replication scheme with the ttCWAdmin -create command, Oracle Clusterware prompts for the user name and password of the TimesTen administration user. If there are cache groups in the active standby pair, Oracle Clusterware also stores the cache manager user name and password.

To change the user name or password for the TimesTen admin user or the cache manager user, perform the following:

1. Disable the Oracle Clusterware management in order to modify the cache manager user name or password with the ttCWAdmin -beginAlterSchema command:

   ttCWAdmin -beginAlterSchema advancedDSN
   

2. Stop the cache agents on both the active and standby masters with either the ttCacheStop built-in procedure or the ttAdmin -cacheStop command.

   ttAdmin -cacheStop cachealone1
   

3. If you want to modify the TimesTen user name or password, use the ALTER USER statement on both the active and standby masters. See "Creating or identifying users to the database" in Oracle TimesTen In-Memory Database Operations Guide.

   Command> ALTER USER cacheuser IDENTIFIED BY newpwd;
   

4. If you want to modify the cache manager user name or password, perform the following:

   Note:

   The cache manager user name must be the same on both TimesTen and the Oracle database; however, the passwords may be different.

   1. On the TimesTen database, if you want to modify the cache manager user name or password, then use the ALTER USER statement on both the active and standby masters. For details on the cache manager user, see "Create users in the TimesTen database" in Oracle In-Memory Database Cache User's Guide.

      Command> ALTER USER cacheuser IDENTIFIED BY newpwd;
      

   2. On the back-end Oracle database, modify the cache manager user name or password with the ALTER USER statement. If you are working on TimesTen, you can use Passthrough 3 to execute this directly on the Oracle database.

      Command> passthrough 3; 
      Command> ALTER USER cacheuser IDENTIFIED BY newpwd;
      

   3. Change the cache manager user name or password on both the active and standby masters by executing the ttCacheUidPwdSet built-in procedure on the TimesTen database. See "Setting the cache manager user name and password" in Oracle In-Memory Database Cache User's Guide.

      Command> call ttCacheUidPwdSet('cacheuser','newpwd');
      

   4. If the user name or password are defined in the sys.odbc.ini file, update the new cache manager user name or password in the sys.odbc.ini file on both the active and standby masters.

5. Start the cache agents on both the active and standby masters with either the ttCacheStart built-in procedure or the ttAdmin -cacheStart command.

   ttAdmin -cacheStart cachealone1
   

6. Re-enable the Oracle Clusterware managment with the new cache administration password with the ttCWAdmin -endAlterSchema command.

   ttCWAdmin -endAlterSchema advancedDSN
   

Obtaining cluster status

Using the -status option of the ttCWAdmin utility reports information about all of the active standby pairs in an instance that are managed by the same instance administrator. If you specify the DSN, the utility reports information for the active standby pair with that DSN.

$ ttCWAdmin -status
TimesTen Cluster status report as of Thu Nov 11 13:54:35 2010
 
====================================================================
TimesTen daemon monitors:
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
====================================================================
TimesTen Cluster agents
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
 
Status of Cluster related to DSN MYDSN:
====================================================================
1. Status of Cluster monitoring components:
Monitor Process for Active datastore:NOT RUNNING
Monitor Process for Standby datastore:NOT RUNNING
Monitor Process for Master Datastore 1 on Host host1: NOT RUNNING
Monitor Process for Master Datastore 2 on Host host2: NOT RUNNING
 
2.Status of  Datastores comprising the cluster
Master Datastore 1:
Host:host1
Status:AVAILABLE
State:ACTIVE
Grid:NO GRID
Master Datastore 2:
Host:host2
Status:UNAVAILABLE
State:UNKNOWN
Grid:UNKNOWN
====================================================================
The cluster containing the replicated DSN is offline

$ ttcwadmin -status
TimesTen Cluster status report as of Thu Nov 11 13:58:25 2010
 
====================================================================
TimesTen daemon monitors:
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
====================================================================
TimesTen Cluster agents
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
 
Status of Cluster related to DSN MYDSN:
====================================================================
1. Status of Cluster monitoring components:
Monitor Process for Active datastore:RUNNING on Host host1
Monitor Process for Standby datastore:RUNNING on Host host1
Monitor Process for Master Datastore 1 on Host host1: RUNNING
Monitor Process for Master Datastore 2 on Host host2: RUNNING
 
2.Status of  Datastores comprising the cluster
Master Datastore 1:
Host:host1
Status:AVAILABLE
State:ACTIVE
Grid:NO GRID
Master Datastore 2:
Host:host2
Status:AVAILABLE
State:IDLE
Grid:NO GRID
====================================================================
The cluster containing the replicated DSN is online

2.Status of  Datastores comprising the cluster
Master Datastore 1:
Host:host1
Status:AVAILABLE
State:ACTIVE
Grid:AVAILABLE
Master Datastore 2:
Host:host2
Status:AVAILABLE
State:IDLE
Grid:NO GRID

$ ttcwadmin -status
TimesTen Cluster status report as of Thu Nov 11 13:59:20 2010
 
====================================================================
TimesTen daemon monitors:
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
====================================================================
TimesTen Cluster agents
Host:HOST1 Status: online
Host:HOST2 Status: online
 
====================================================================
 
Status of Cluster related to DSN MYDSN:
====================================================================
1. Status of Cluster monitoring components:
Monitor Process for Active datastore:RUNNING on Host host1
Monitor Process for Standby datastore:RUNNING on Host host2
Monitor Process for Master Datastore 1 on Host host1: RUNNING
Monitor Process for Master Datastore 2 on Host host2: RUNNING
 
2.Status of  Datastores comprising the cluster
Master Datastore 1:
Host:host1
Status:AVAILABLE
State:ACTIVE
Grid:NO GRID
Master Datastore 2:
Host:host2
Status:AVAILABLE
State:STANDBY
Grid:NO GRID
====================================================================
The cluster containing the replicated DSN is online

2.Status of  Datastores comprising the cluster
Master Datastore 1:
Host:host1
Status:AVAILABLE
State:ACTIVE
Grid:AVAILABLE
Master Datastore 2:
Host:host2
Status:AVAILABLE
State:STANDBY
Grid:AVAILABLE

Message log files

The monitor processes report events and errors to the ttcwerrors.log and ttcwmsg.log files. The files are located in the daemon_home/info directory. The default size of these files is the same as the default maximum size of the user log. The maximum number of log files is the same as the default number of files for the user log. When the maximum number of files has been written, additional errors and messages overwrite the files, beginning with the oldest file.

For the default values for number of log files and log file size, see "Modifying informational messages" in Oracle TimesTen In-Memory Database Operations Guide.