10 Troubleshooting Data Guard

10.2.1 ORA-16596: Object Not Part of the Data Guard Broker Configuration

A request was issued to the broker, but the database instance through which you have connected is no longer a part of the broker configuration. You may see this error when the broker monitor (DMON) fails to locate a broker configuration profile for the database upon which it is running.

Solution Reconnect to the configuration through another database instance. Confirm that the value of the DB_UNIQUE_NAME initialization parameter for the database and the database object name in the broker configuration are identical.

This problem can also occur if you attempt to enable a configuration, but the broker configuration file for one of its databases was accidentally removed or is outdated. In this case, remove the database from the broker configuration, manually delete the configuration file for that standby database (not for the primary database), and try again to enable the configuration. After the configuration is enabled, run the Add Standby Database wizard and choose the Add existing standby database option to create a new database profile for the previously deleted standby database.

10.2.2 Redo Accumulating on the Primary Is Not Sent to Some Standby Databases

By viewing the Log File Details page in Enterprise Manager, you have determined that log files are accumulating on the primary database and are not being archived to some of the standby databases in the broker configuration.

Solution To narrow down the problem, do the following:

• Verify that the state of the primary database is ONLINE (not LOG-TRANSPORT-OFF).

• Verify that the value of the LogShipping property of the standby database in question is ON.

• Check the status of the redo transport services on the primary database using the LogXptStatus monitorable property. If redo transport services have an error, then use the error message to determine further checking and resolution action. For example:

  • If the error indicates the standby database is not available, you need to restart the standby database.

  • If the error indicates no listener, you need to restart the listener.

  • If the error indicates the standby database has no local destination, you need to set up a standby location to store archived redo log files from the primary database.

10.2.3 Many Log Files Are Received on a Standby Database But Not Applied

By viewing the Performance page or Log File Details page in Enterprise Manager, you have determined that the standby database accumulates too many log files without applying them.

Solution There are many possible reasons why archived redo log files might not be applied to the standby database. Investigate why the log files are building up and rule out the valid reasons.

If the current status of the standby database is not normal:

• Determine whether or not the log apply services might be unexpectedly offline. See the ORA-16766 (for physical standby databases) or ORA-16768 (for logical standby databases) problem and solution statement for more help.

• If this is a logical standby database, check to see if a failed transaction has occurred.

• If you want to suppress the error while you investigate the problem, you can temporarily disable broker management of the database.

  See Also:

  Chapter 8 for additional information about disabling the database using the DGMGRL command-line interface

If the current status of the standby database is normal:

• Verify the state of the standby database is online (not in the LOG-APPLY-OFF, OFFLINE, or READ-ONLY states).

• Verify the state of the primary database is online (not in the LOG-TRANSPORT-OFF state).

• See Also:

  Chapter 9 for additional information about the LogShipping property

• Check to see if log files are building up because the value of the DelayMins property is set too large. (Log apply services will delay applying the archived redo log files on the standby database for the number of minutes specified.)

  See Also:

  Chapter 9 for additional information about the DelayMins property

• If you cannot see any errors, compare the archive rate to the apply rate on the Performance page in Enterprise Manager to see if the apply rate is lower than the archive rate.

10.2.4 The Request Timed Out or Enterprise Manager Performance Is Sluggish

If the broker requests are not completing within the normal timeout parameters, try the following actions to solve the problem:

1. Verify the network is operating appropriately.

2. Try to ping all of the nodes in the configuration.

3. Try reconnecting through another database to retry the operation.

4. Run the VERIFY command from the configuration to see which broker is busy. The output from the VERIFY command should show why (or at least which) broker is not able to process the requests.

10.2.5 The Primary Database is Flashed Back

If the primary database is flashed back, the standby databases in the configuration must be also be flashed back or reenabled to be viable targets for switchovers or failovers. The broker will report errors for the standby databases if the primary database has been flashed back.

For more information about restoring the viability of a standby database that was disabled by the broker, see Section 5.4.3.

10.5.1 Problems Starting the Observer

Only one observer can be associated with the broker configuration at any given time. If you attempt to start a second observer, one of the following errors is returned:

ORA-16647: could not start more than one observer
DGM-16954: Unable to open and lock the Observer configuration file

Use the DGMGRL SHOW CONFIGURATION VERBOSE command to determine the location of the observer that is currently associated with the broker configuration.

DGMGRL> SHOW CONFIGURATION VERBOSE;
Configuration
  Name:                DRSolution
  Enabled:             YES
  Protection Mode:     MaxAvailability
  Fast-Start Failover: ENABLED
  Databases:
    North_Sales - Primary database
    DR_Sales    - Physical standby database
                - Fast-Start Failover target
 
Fast-Start Failover
  Threshold: 45 seconds
  Observer:  observer.foo.com
 
Current status for "DRSolution":
SUCCESS

10.5.2 Problems Because the Observer Has Stopped

If the observer host machine crashes, the broker configuration is no longer observed and fast-start failover is no longer possible. In this case, you may have to move the observer to a new host if the original host machine cannot be repaired in a timely fashion. To move the observer, you must stop the first observer and then start a new observer on another host.

1. Issue the DGMGRL STOP OBSERVER command to sever the link between the original observer and the broker configuration:

   DGMGRL> STOP OBSERVER;
   Done.
   
   

2. Issue the DGMGRL SHOW CONFIGURATION VERBOSE and the SHOW DATABASE command to verify that the configuration is no longer being observed:

   DGMGRL> SHOW CONFIGURATION VERBOSE;
    
   Configuration
     Name:                DRSolution
     Enabled:             YES
     Protection Mode:     MaxAvailability
     Fast-Start Failover: ENABLED
     Databases:
       North_Sales - Primary database
       DRSales     - Physical standby database
                   - Fast-Start Failover target
    
   Fast-Start Failover
     Threshold: 45 seconds
     Observer:  (none)
    
   Current status for "The SUPER cluster":
   Warning: ORA-16608: one or more databases have warnings
   
   DGMGRL> SHOW DATABASE 'North_Sales';
   Database
     Name:            North_Sales
     Role:            PRIMARY
     Enabled:         YES
     Intended State:  ONLINE
     Instance(s):
       sales1
   Current status for "North_Sales":
   Error: ORA-16658: unobserved Fast-Start Failover configuration
   
   

3. Note that you do not need to issue the DGMGRL SHOW commands to verify that the observer has actually stopped. Successful completion of the DGMGRL STOP OBSERVER command will allow a new observer to become associated with the configuration.

   Now, you may start the new observer. After the observer is started, it prints its failover and reinstatement actions to the console in which it was started. For example:

   DGMGRL> START OBSERVER;
   Observer started
    
   14:41:23.95  Thursday, May 26, 2005
   Initiating fast-start failover to database "DRSales"...
   Performing failover NOW, please wait...
   Failover succeeded, new primary is "DRSales"
   14:41:38.62  Thursday, May 26, 2005
    
   14:42:15.43  Thursday, May 26, 2005
   Initiating reinstatement for database "North_Sales"...
   Reinstating database "North_Sales", please wait...
   Operation requires shutdown of instance "sales1" on database "North_Sales"
   Shutting down instance "sales1"...
   ORA-01109: database not open
    
   Database dismounted.
   ORACLE instance shut down.
   Operation requires startup of instance "sales1" on database "North_Sales"
   Starting instance "sales1"...
   ORACLE instance started.
   Database mounted.
   Continuing to reinstate database "North_Sales" ...
   Reinstatement of database "North_Sales" succeeded
   14:43:23.71  Thursday, May 26, 2005
   

10.5.3 Capturing Observer Actions in the Observer Log File

You can use the DGMGRL -logfile option to start the observer, so that all of the troubleshooting actions performed in Section 10.5.1 can be captured in a file. For example:

% dgmgrl -logfile observer.log / "start observer"

All the observer output is then recorded in a file named observer.log in the current working directory where you issued the DGMGRL command.