3 Monitoring Using Oracle Grid Control

3.2.1 Set Up Default Notification Rules for Each System

Notification Rules are defined sets of alerts on metrics that are automatically applied to a target when it is discovered by Oracle Grid Control. For example, an administrator can create a rule that monitors the availability of database targets and generates an e-mail message if a database fails. After that rule is generated, it is applied to all existing databases and any database created in the future. Access these rules by navigating to Preferences and then choosing Rules.

The rules monitor problems that require immediate attention, such as those that can affect service availability, and Oracle or application errors. Service availability can be affected by an outage in any layer of the application stack: node, database, listener, and critical application data. A service availability failure, such as the inability to connect to the database, or the inability to access data critical to the functionality of the application, must be identified, reported, and reacted to quickly. Potential service outages such as a full archive log directory also must be addressed correctly to avoid a system outage.

Oracle Grid Control provides a series of default rules that provide a strong framework for monitoring availability. A default rule is provided for each of the preinstalled target types that come with Oracle Grid Control. These rules can be modified to conform to the policies of each individual site, and new rules can be created for site-specific targets or applications. The rules can also be set to notify users during specific time periods to create an automated coverage policy.

Consider the following recommendations:

• Modify each rule for high-value components in the target architecture to suit the required availability requirements by using the rules modification wizard. For the database rule, set the events in Table 3-1, Table 3-2, and Table 3-3 for each target. The frequency of the monitoring is determined by the service level agreement (SLA) for each component.

• Use Beacon functionality to track the performance of individual applications. A Beacon can be set to perform a user transaction representative of normal application work. Enterprise Manager can then break down the response time of that transaction into its component pieces for analysis. In addition, an alert can be triggered if the execution time of that transaction exceeds a predefined limit.

  See Also:

  • Oracle Enterprise Manager Concepts for conceptual information about Beacons

  • Oracle Enterprise Manager Advanced Configuration for information about configuring service tests and Beacons

• Add Notification Methods and use them in each Notification Rule. By default, the easiest method for alerting an administrator to a potential problem is to send e-mail. Supplement this notification method by adding a callout to an SNMP trap or operating system script that sends an alert by some method other than e-mail. This avoids the problem that might occur if a component of the e-mail system has failed. Set additional Notification Methods by using the Set-up link at the top of any Oracle Grid Control page.

• Modify Notification Rules to notify the administrator when there are errors in computing target availability. This might generate a false positive reading on the availability of the component, but it ensures the highest level of notification to system administrators.

Figure 3-2 shows the Notification Rule property page for choosing availability states with Down, Agent Unreachable, Agent Unreachable Resolved, and Metric Error Detected chosen.

Figure 3-2 Setting Notification Rules for Availability

Description of "Figure 3-2 Setting Notification Rules for Availability"

In addition, modify the metrics monitored by the database rule to report the metrics shown in Table 3-1, Table 3-2, and Table 3-3. This ensures that these metrics are captured for all database targets and that trend data will be available for future analysis. All events described in Table 3-1, Table 3-2, and Table 3-3 can be accessed from the Database Homepage by choosing All Metrics > Expand All.

Space management conditions that have the potential to cause a service outage should be monitored using the events shown in Table 3-1.

From the Alert Log Metric group, set Oracle Grid Control to monitor the alert log for errors as shown in Table 3-2.

Monitor the system to ensure that the processing capacity is not exceeded. The warning and critical levels for these events should be modified based on the usage pattern of the system. Set the events from the Database Limits metric group using the recommendations in Table 3-3.

Figure 3-3 shows the Notification Rule property page for setting choosing metrics. The user has chosen Critical and Warning as the severity states for notification. The list of Available Metrics is shown in the left list box. The metrics that have been selected for notification are shown in the right list box.

Figure 3-3 Setting Notification Rules for Metrics

Description of "Figure 3-3 Setting Notification Rules for Metrics"

3.2.2 Use Database Target Views to Monitor Health, Availability, and Performance

The Database Targets page in Figure 3-4 shows an overview of system performance, space utilization, and the configuration of important availability components like archived redo log status, flashback log status, and estimated instance recovery time. Alerts are displayed immediately. Each of the alert values can be configured from links on this page.

Figure 3-4 Overview of System Performance

Description of "Figure 3-4 Overview of System Performance"

Many of the metrics from the Oracle Grid Control pertain to performance. A system without adequate performance is not an HA system, regardless of the status of any individual component. While performance problems seldom cause a major system outage, they can still cause an outage to a subset of customers. Outages of this type are commonly referred to as application service brownouts. The primary cause of brownouts is the intermittent or partial failure of one or more infrastructure components. IT managers must be aware of how the infrastructure components are performing (their response time, latency, and availability), and how they are affecting the quality of application service delivered to the end user.

A performance baseline, derived from normal operations that meet the SLA, should determine what constitutes a performance metric alert. Baseline data should be collected from the first day that an application is in production and should include the following:

• Application statistics (transaction volumes, response time, Web service times)

• Database statistics (transaction rate, redo rate, hit ratios, top 5 wait events, top 5 SQL transactions)

• Operating system statistics (CPU, memory, I/O, network)

You can use Oracle Grid Control to capture a snapshot of database performance as a baseline. Oracle Grid Control compares these values against system performance and displays the result on the database Target page. It can also send alerts if the values deviate too far from the established baseline.

Set the database notification rule to capture the metrics listed in Table 3-4 for all database targets. Analysis of these parameters can then be done using one tool and historical data will be available.

3.2.3 Use Event Notifications to React to Metric Changes

There are many operating system events that can be used to supplement a suggested metric. Such operating system events are not required for each host and instance. All metrics defined here can be set individually by instance or database using the Manage Metrics link at the bottom of the navigation bar on the object target page. The values that trigger a warning or critical alert can be changed here, and an operating system script can be activated to respond to an metric threshold, in addition to the standard alert being generated to the Oracle Grid Control.

3.2.4 Use Events to Monitor Data Guard System Availability

Set Oracle Grid Control metrics to monitor the availability of logical and physical Data Guard configurations. If a Data Guard environment is registered with the Data Guard Manager extension of Oracle Grid Control, then set the events shown in Table 3-5.

3.3.1 Check Oracle Grid Control Policy Violations

Oracle Grid Control comes with a pre-installed set of policies and recommendations of best practices for all databases. These policies are checked by default, and the number of violations is displayed on the Targets page shown in Figure 3-4. To see a list of all violations, select Policy Violations from the Targets page.

3.3.2 Use Oracle Grid Control to Manage Oracle Patches and Maintain System Baselines

You can use Oracle Grid Control to download and manage patches from https://metalink.oracle.com for any monitored system in the application environment. A job can be set up to routinely check for patches that are relevant to the user environment. Those patches can be downloaded and stored directly in the Management Repository. Patches can be staged from the Management Repository to multiple systems and applied during maintenance windows.

You can examine patch levels for one machine and compare them between machines in either a one-to-one or one-to-many relationship. In this case, a machine can be identified as a baseline and used to demonstrate maintenance requirements in other machines. This can be done for operating system patches as well as database patches.

3.3.3 Use Oracle Grid Control to Manage Data Guard Targets

Oracle Grid Control can be used to set up logical and physical standby databases for any database target. It also provides the ability to manage switchover and failover of database targets other than the database that contains the Management Repository.

Oracle Grid Control can also be used to monitor the health of a Data Guard configuration at a glance. From any database target page, navigate to the Data Guard status section by using the link in the High Availability section. The page shows the active standby databases for the primary target, the amount of log data waiting for shipment and receipt by the standby database, and the data protection mode. You can also modify the data protection mode from this page.

This page contains a link to the Verify function, which checks the Data Guard environment and redo transport services to display warnings and errors. The Verify function is not automatic and must be run manually.