H Troubleshooting Oracle Clusterware

CHM Services

CHM consists of the following services:

• System Monitor Service

• Cluster Logger Service

System Monitor Service

There is one system monitor service on every node. The system monitor service (osysmond) is the monitoring and operating system metric collection service that sends the data to the cluster logger service. The cluster logger service receives the information from all the nodes and persists in a CHM repository-based database.

Cluster Logger Service

There is one cluster logger service (ologgerd) on only one node in a cluster and another node is chosen by the cluster logger service to house the standby for the master cluster logger service. If the master cluster logger service fails (because the service is not able come up after a fixed number of retries or the node where the master was running is down), the node where the standby resides takes over as master and selects a new node for standby. The master manages the operating system metric database in the CHM repository and interacts with the standby to manage a replica of the master operating system metrics database.

CHM Repository

The CHM repository, by default, resides within the Grid Infrastructure home and requires 1 GB of disk space per node in the cluster. You can adjust its size and location, and Oracle supports moving it to shared storage. You manage the CHM repository with OCLUMON.

Collecting CHM Data

You can collect CHM data from any node in the cluster by running the Grid_home/bin/diagcollection.pl script on the node.

To run the data collection script on only the node where the cluster logger service is running:

1. Run the following command to identify the node running the cluster logger service:

   $ Grid_home/bin/oclumon manage -get master
   

2. Run the following command as a privileged user on the cluster logger service node to collect all the available data in the Grid Infrastructure Management Repository:

   # Grid_home/bin/diagcollection.pl
   

   The diagcollection.pl script creates a file called chmosData_host_name_time_stamp.tar.gz, similar to the following:

   chmosData_stact29_20121006_2321.tar.gz
   

To limit the amount of data you want collected:

# Grid_home/bin/diagcollection.pl -collect -chmos
   -incidenttime inc_time -incidentduration duration

In the preceding command, the format for the -incidenttime parameter is MM/DD/YYYY24HH:MM:SS and the format for the -incidentduration parameter is HH:MM. For example:

# Grid_home/bin/diagcollection.pl -collect -crshome
   Grid_home -chmoshome Grid_home -chmos -incidenttime 07/14/201201:00:00
   -incidentduration 00:30

oclumon debug

Use the oclumon debug command to set the log level for the CHM services.

Syntax

oclumon debug [log daemon module:log_level] [version]

Parameters

log daemon module:log_level

version

Example

The following example sets the log level of the system monitor service (osysmond):

$ oclumon debug log osysmond CRFMOND:3

oclumon dumpnodeview

Use the oclumon dumpnodeview command to view log information from the system monitor service in the form of a node view.

A node view is a collection of all metrics collected by CHM for a node at a point in time. CHM attempts to collect metrics every second on every node. Some metrics are static while other metrics are dynamic.

A node view consists of seven views when you display verbose output:

• SYSTEM: Lists system metrics such as CPU COUNT, CPU USAGE, and MEM USAGE

• TOP CONSUMERS: Lists the top consuming processes in the following format:

  metric_name: 'process_name(process_identifier) utilization'
  

• PROCESSES: Lists process metrics such as PID, name, number of threads, memory usage, and number of file descriptors

• DEVICES: Lists device metrics such as disk read and write rates, queue length, and wait time per I/O

• NICS: Lists network interface card metrics such as network receive and send rates, effective bandwidth, and error rates

• FILESYSTEMS: Lists file system metrics, such as total, used, and available space

• PROTOCOL ERRORS: Lists any protocol errors

You can generate a summary report that only contains the SYSTEM and TOP CONSUMERS views.

"Metric Descriptions" lists descriptions for all the metrics associated with each of the views in the preceding list.

Example H-1 shows an example of a node view.

Syntax

oclumon dumpnodeview [[-allnodes] | [-n node1 node2] [-last "duration"] | 
[-s "time_stamp" -e "time_stamp"] [-v] [-warning]] [-h]

Parameters

-allnodes

-n node1 node2

-last "duration"

"23:05:00"

-s "time_stamp" -e "time_stamp"

"2011-05-10 23:05:00"

-v

-warning

-h

Usage Notes

• The default is to continuously dump node views. To stop continuous display, use Ctrl+C on Linux and Esc on Windows.

• Both the local system monitor service (osysmond) and the cluster logger service (ologgerd) must be running to obtain node view dumps.

Examples

The following example dumps node views from node1, node2, and node3 collected over the last twelve hours:

$ oclumon dumpnodeview -n node1 node2 node3 -last "12:00:00"

The following example displays node views from all nodes collected over the last fifteen minutes:

$ oclumon dumpnodeview -allnodes -last "00:15:00"

Metric Descriptions

This section includes descriptions of the metrics in each of the seven views that make up a node view listed in the following tables.

#pcpus

#vcpus

chipname

cpuht

cpu

cpuq

physmemfree

physmemtotal

mcache

swapfree

swaptotal

ior

iow

ios

swpin

swpout

pgin

pgout

netr

netw

procs

rtprocs

#fds

#sysfdlimit

#disks

#nics

nicErrors

name

pid

#procfdlimit

cpuusage

memusage

shm

workingset

#fd

#threads

priority

nice

ior

iow

ios

qlen

wait

type

netrr

netwr

neteff

nicerrors

pktsin

pktsout

errsin

errsout

indiscarded

outdiscarded

inunicast

type

innonunicast

latency

total

used

available

used%

mft%

ifree%

IPHdrErr

IPAddrErr

IPUnkProto

IPReasFail

IPFragFail

TCPFailedConn

TCPEstRst

TCPRetraSeg

UDPUnkPort

UDPRcvErr

^Footnote 1 All protocol errors are cumulative values since system startup.

----------------------------------------
Node: node1 Clock: '07-17-13 23.33.25' SerialNo:34836
----------------------------------------

SYSTEM:
#pcpus: 12 #vcpus: 12 cpuht: N chipname: Intel(R) cpu: 2.43 cpuq: 2
physmemfree: 56883116 physmemtotal: 74369536 mcache: 13615352 swapfree: 18480408
swaptotal: 18480408 ior: 170 iow: 37 ios: 37 swpin: 0 swpout: 0 pgin: 170
pgout: 37 netr: 40.301 netw: 57.211 procs: 437 rtprocs: 33 #fds: 15008
#sysfdlimit: 6815744 #disks: 9 #nics: 5  nicErrors: 0

TOP CONSUMERS:
topcpu: 'osysmond.bin(9103) 2.59' topprivmem: 'java(26616) 296808'
topshm: 'ora_mman_orcl_4(32128) 1222220' topfd: 'ohasd.bin(7594) 150'
topthread: 'crsd.bin(9250) 43'

PROCESSES:

name: 'mdnsd' pid: 12875 #procfdlimit: 8192 cpuusage: 0.19 privmem: 9300
shm: 8604 #fd: 36 #threads: 3 priority: 15 nice: 0
name: 'ora_cjq0_rdbms3' pid: 12869 #procfdlimit: 8192 cpuusage: 0.39
privmem: 10572 shm: 77420 #fd: 23 #threads: 1 priority: 15 nice: 0
name: 'ora_lms0_rdbms2' pid: 12635 #procfdlimit: 8192 cpuusage: 0.19
privmem: 15832 shm: 49988 #fd: 24 #threads: 1 priority: 15 nice: 0
name: 'evmlogger' pid: 32355 #procfdlimit: 8192 cpuusage: 0.0 privmem: 4600
shm: 8756 #fd: 9 #threads: 3 priority: 15 nice: 0
.
.
.

DEVICES:

xvda ior: 0.798 iow: 193.723 ios: 33 qlen: 0 wait: 0 type: SWAP
xvda2 ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SWAP
xvda1 ior: 0.798 iow: 193.723 ios: 33 qlen: 0 wait: 0 type: SYS

NICS:

lo netrr: 35.743  netwr: 35.743  neteff: 71.486  nicerrors: 0 pktsin: 22
pktsout: 22  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0
inunicast: 22 innonunicast: 0  type: PUBLIC
eth0 netrr: 7.607  netwr: 1.363  neteff: 8.971  nicerrors: 0 pktsin: 41  
pktsout: 18  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0  
inunicast: 41  innonunicast: 0  type: PRIVATE latency: <1

FILESYSTEMS:

mount: / type: rootfs total: 155401100 used: 125927608 available: 21452240
used%: 85 ifree%: 93 [ORACLE_HOME CRF_HOME rdbms2 rdbms3 rdbms4 has51]

mount: /scratch type: ext3 total: 155401100 used: 125927608 available: 21452240
used%: 85 ifree%: 93 [rdbms2 rdbms3 rdbms4 has51]

mount: /net/adc6160173/scratch type: ext3 total: 155401100 used: 125927608
 available: 21452240 used%: 85 ifree%: 93 [rdbms2 rdbms4 has51]

PROTOCOL ERRORS:

IPHdrErr: 0 IPAddrErr: 19568 IPUnkProto: 0 IPReasFail: 0 IPFragFail: 0
TCPFailedConn: 931776 TCPEstRst: 76506 TCPRetraSeg: 12258 UDPUnkPort: 29132
UDPRcvErr: 148

oclumon manage

Use the oclumon manage command to view log information from the system monitor service.

Syntax

oclumon manage [[-repos {resize size | changesize memory_size | 
reploc new_location [[-maxtime size] | [-maxspace memory_size]]}] | 
[-get key1 key2 ...]]

Parameters

-repos {resize size | changesize memory_size | 
reploc new_location
 [[-maxtime size | 
 -maxspace memory_size]]

-get key1 key2 ...

-h

Usage Notes

Both the local system monitor service and the master cluster logger service must be running to resize the CHM repository.

Example

The following examples show commands and sample output:

$ oclumon manage -repos reploc /shared/oracle/chm

The preceding example moves the CHM repository to shared storage.

$ oclumon manage -get reppath
CHM Repository Path = /opt/oracle/grid/crf/db/node1
Done

$ oclumon manage -get master
Master = node1
done

$ oclumon manage -get repsize
CHM Repository Size = 86400
Done

oclumon version

Use the oclumon version command to obtain the version of CHM that you are using.

Syntax

oclumon version

Example

This command produces output similar to the following:

Cluster Health Monitor (OS), Version 11.2.0.3.0 - Production Copyright 2011
Oracle. All rights reserved.

tfactl print

Use the tfactl print command to print information from the Berkeley database.

Syntax

tfactl print [status | config | directories | hosts | actions | repository | cookie]

Parameters

status

config

directories

hosts

actions

repository

cookie

Example

The tfactl print config returns output similar to the following:

Configuration parameter                                Value
------------------------------------------------------------
TFA Version                                          2.5.1.5
Automatic diagnostic collection                          OFF
Trimming of files during diagcollection                   ON
Repository current size (MB) in node1                    526
Repository maximum size (MB) in node1                  10240
Trace Level                                                1

In the preceding sample output:

• Automatic diagnostic collection: When ON (default is OFF), if scanning an alert log, then finding specific events in those logs triggers diagnostic collection.

• Trimming of files during diagcollection: Determines if TFA will trim large files to contain only data that is within specified time ranges. When this is OFF, no trimming of trace files occurs for automatic diagnostic collection.

• Repository current size in MB: How much space in the repository is currently used.

• Repository maximum size in MB: The maximum size of storage space in the repository. Initially, the maximum size is set to the smaller of either 10GB or 50% of free space in the file system.

• Trace Level: 1 is the default, and the values 2, 3, and 4 have increasing verbosity. While you can set the trace level dynamically for the running TFA daemon, increasing the trace level significantly impacts the performance of TFA, and should only be done at the request of My Oracle Support.

tfactl purge

Use the tfactl purge command to delete diagnostic collections from the TFA repository that are older than a specific time.

Syntax

tfactl purge -older number[h | d]

Specify a number followed by h or d (to specify a number of hours or days, respectively) to remove files older than the specified time constraint. For example:

# tfactl purge -older 30d

The preceding command removes files older than 30 days.

tfactl directory

Use the tfactl directory command to add a directory to, or remove a directory from, the list of directories that will have their trace or log files analyzed. You can also use this command to change the directory permissions. When a directory is added by auto discovery, it is added as public, which means that any file in that directory can be collected by any user that has permission to run the tfactl diagcollect command. This is only important when sudo is used to allow users other than root to run TFACTL commands. If a directory is marked as private, then TFA determines which user is running TFACTL commands and verifies that the user has permissions to see the files in the directory before allowing any files to be collected.

Syntax

tfactl directory [add directory | remove directory | modify directory -private | -public]

Parameters

add directory

remove directory

modify directory
-private | -public

Usage Notes

You must add all trace directory names to the Berkeley database, so that TFA will collect file metadata in that directory. The discovery process finds most directories but if new or undiscovered directories are required, then you can add these manually using the tfactl directory command. When you add a directory using TFACTL, TFA attempts to determine whether the directory is for the database, Oracle Clusterware, operating system logs, or some other component, and for which database or instance. If TFA cannot determine this information, then it returns an error and requests that you enter the information, similar to the following:

# tfactl directory add /tmp

Failed to add directory to TFA. Unable to determine parameters for directory: /tmp
Please enter component for this Directory [RDBMS|CRS|ASM|INSTALL|OS|CFGTOOLS] : RDBMS
Please enter database name for this Directory :MYDB
Please enter instance name for this Directory :MYDB1

tfactl host

Use the tfactl host command to add hosts to, or remove hosts from, the TFA cluster.

Syntax

tfactl host [add host_name | remove host_name]

Specify a host name to add or remove, as in the following example:

# tfactl host add myhost.domain.com

Usage Notes

Using the tfactl host add command notifies the local TFA about other nodes on your network. When you add a host, TFA contacts that host and, if TFA is running on that host, then both hosts synchronize their host list. TFA authenticates that a host can be added using a cookie. If the host to be added does not have the correct cookie, then you must retrieve that cookie from an existing host in the cluster and set it on the host being added, similar to the following:

#tfactl host add node2

Failed to add host: node2 as the TFA cookies do not match.
To add the host successfully, try the following steps:
1. Get the cookie in node1 using:
./tfa_home/bin/tfactl print cookie
2. Set the cookie from Step 1 in node2 using:
  ./tfa_home/bin/tfactl set cookie=<COOKIE>
3. After Step 2, add host again:
 ./tfa_home/bin/tfactl host add node2

After you successfully add the host, all clusterwide commands will activate on all nodes registered in the Berkeley database.

tfactl set

Use the tfactl set command to adjust the manner in which TFA runs.

Syntax

tfactl set [autodiagcollect=ON | OFF | cookie=Xn | trimfiles=ON | OFF |
tracelevel=1 | 2 | 3 | 4 | reposizeMB=number | repositorydir=directory] [-c]

Parameters

autodiagcollect=ON | OFF

cookie=UID

trimfiles=ON | OFF

tracelevel=1 | 2 | 3 | 4

-c

Automatic Diagnostic Collection

After TFA initially gathers trace file metadata, the daemon monitors all files that are determined to be alert logs using tail, so that TFA can take action when certain strings occur.

By default, these logs are database alert logs, Oracle ASM alert logs, and Oracle Clusterware alert logs. When specific patterns occur in the logs saved to the Berkeley database, automatic diagnostic collection may take place.

Exactly what is collected depends on the pattern that is matched. TFA may just store information on the pattern matched or may initiate local diagnostic collection. Although TFA always monitors the logs and collects information into its database, automatic diagnostic collection only happens if it is enabled first using the tfactl set command.

tfactl diagcollect

Use the tfactl diagcollect command to perform on-demand diagnostic collection. You can configure a number of different parameters to determine how large or detailed a collection you want. You can specify a specific time of an incident or a time range for data to be collected, and determine if whole files that have relevant data should be collected or just a time interval of data from those files.

Syntax

tfactl diagcollect [-all | -database all | database_1,database_2,... | -asm | -crs | -os | -install]
[-node all | local | node_1,node_2,...][-tag description] [-z file_name]
[-since numberh | d | -from "mmm/dd/yyyy hh:mm:ss" -to "mmm/dd/yyyy hh:mm:ss"
| -for "mmm/dd/yyyy hh:mm:ss" [-nocopy] [-nomonitor]]

Parameters

-all | -database all
 | database_1,database_2,...
 | -asm | -crs | -os |
 -install

-node all | local | node_1,node_2,...

-tag description

-z file_name

-since numberh | d |
 -from "mmm/dd/yyyy hh:mm:ss" 
-to "mmm/dd/yyyy hh:mm:ss" |
 -for "mmm/dd/yyyy hh:mm:ss"

-nocopy

-nomonitor

Examples

• The following command trims and zips all files updated in the last four hours, including chmos and osw data, from across the cluster and collects it on the initiating node:

  # tfactl diagcollect
  

• The following command trims and zips all files updated in the last eight hours, including chmos and osw data, from across the cluster and collects it on the initiating node:

  # tfactl diagcollect –all –since 8h
  

• The following command trims and zips all files from databases hrdb and fdb updated in the last one day and collects it on the initiating node:

  # tfactl diagcollect -database hrdb,fdb -since 1d -z foo
  

• The following command trims and zips all Oracle Clusterware files, operating system logs, and chmos and osw data from node1 and node2 updated in the last six hours, and collects it on the initiating node.

  # tfactl diagcollect –crs -os -node node1,node2 -since 6h
  

• The following command trims and zips all Oracle ASM logs from node1 updated between July 4, 2013 and July 5, 2013 at 21:00, and collects it on the initiating node:

  # tfactl diagcollect -asm -node node1 -from Jul/4/2013 -to "Jul/5/2013 21:00:00"
  

• The following command trims and zips all log files updated on July 2, 2013 and collect at the initiating node:

  # tfactl diagcollect -for Jul/2/2013
  

• The following command trims and zips all log files updated from 09:00 on July 2, 2013, to 09:00 on July 3, 2013, which is 12 hours before and after the time specified in the command, and collects it on the initiating node:

  # tfactl diagcollect -for "Jul/2/2013 21:00:00"