A Troubleshooting Enterprise Manager

Prerequisite Check Fails with Directories Not Empty Error During Retry

During an agent installation using Agent Deploy, the installation fails abruptly, displaying the Failure page. On clicking Retry, the installation fails again at the Prerequisite Check phase with an error stating the directories are not empty.

This could be because Oracle Universal Installer (OUI) is still running though the SSH connection that is closed on the remote host.

To resolve this issue, on the remote host, check if OUI is still running. Execute the following command to verify this:

ps -aef | grep -i ora

If OUI is still running, wait till OUI processes are complete and restart the SSH daemon. Now, you can click Retry to perform the installation.

Agent Deployment on Linux Oracle RAC 10.2 Cluster Fails

Agent deployment on a 10.2 release of an Oracle RAC cluster may fail due to a lost SSH connection during the installation process.

This can happen if the LoginGraceTime value in the sshd_config file is 0 (zero). The zero value gives an indefinite time for SSH authentication.

To resolve this issue, modify the LoginGraceTime value in the /etc/ssh/sshd_config file be a higher value. The default value is 120 seconds. This means that the server will disconnect after this time if you have not successfully logged in.

To resolve this issue, modify the LoginGraceTime value in the /etc/ssh/sshd_config file to be a higher value. If the value is set to 0 (zero), there is no definite time limit for authentication.

SSH User Equivalence Verification Fails During Agent Installation

The most common reasons for SSH User Equivalence Verification to fail are the following:

• The server settings in /etc/sshd/sshd_config file do not allow ssh for user $USER.

• The server may have disabled the public key-based authentication.

• The client public key on the server may be outdated.

• You may not have passed the -shared option for shared remote users, or may have passed this option for non-shared remote users.

Verify the server setting and rerun the script to set up SSH User Equivalence successfully.

#    $OpenBSD: sshd_config,v 1.59 2002/09/25 11:17:16 markus Exp $# This is the sshd server system-wide configuration file.  See# sshd_config(5) for more information.# This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin# The strategy used for options in the default sshd_config shipped with# OpenSSH is to specify options with their default values where# possible, but leave them commented out.  Uncommented options change a# default value.#Port 22#Protocol 2,1#ListenAddress 0.0.0.0#ListenAddress ::# HostKey for protocol version 1#HostKey /etc/ssh/ssh_host_key# HostKeys for protocol version 2#HostKey /etc/ssh/ssh_host_rsa_key#HostKey /etc/ssh/ssh_host_dsa_key# Lifetime and size of ephemeral version 1 server key#KeyRegenerationInterval 3600#ServerKeyBits 768# Logging#obsoletes QuietMode#SyslogFacility AUTHSyslogFacility AUTHPRIV
#LogLevel INFO

# Authentication:#LoginGraceTime 120#PermitRootLogin yes#StrictModes yes

#RSAAuthentication yes#PubkeyAuthentication yes#AuthorizedKeysFile       .ssh/authorized_keys# rhosts authentication should not be used#RhostsAuthentication no# Don't read the user's ~/.rhosts and ~/.shosts files#IgnoreRhosts yes# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts#RhostsRSAAuthentication no# similar for protocol version 2#HostbasedAuthentication no# Change to yes if you don't trust ~/.ssh/known_hosts for# RhostsRSAAuthentication and HostbasedAuthentication#IgnoreUserKnownHosts no# To disable tunneled clear text passwords, change to no here!#PasswordAuthentication yes#PermitEmptyPasswords no# Change to no to disable s/key passwords#ChallengeResponseAuthentication yes# Kerberos options#KerberosAuthentication no#KerberosOrLocalPasswd yes#KerberosTicketCleanup yes#AFSTokenPassing no# Kerberos TGT Passing only works with the AFS kaserver#KerberosTgtPassing no# Set this to 'yes' to enable PAM keyboard-interactive authentication# Warning: enabling this may bypass the setting of 'PasswordAuthentication'#PAMAuthenticationViaKbdInt no#X11Forwarding noX11Forwarding yes#X11DisplayOffset 10#X11UseLocalhost yes#PrintMotd yes#PrintLastLog yes
#KeepAlive yes
#UseLogin no#UsePrivilegeSeparation yes#PermitUserEnvironment no#Compression yes#MaxStartups 10# no default banner path#Banner /some/path#VerifyReverseMapping no#ShowPatchLevel no# override default of no subsystemsSubsystem       sftp    /usr/libexec/openssh/sftp-server

SSH Setup Fails with "Invalid Port Number" Error

The SSH User Equivalence script when executed, is built to automatically verify the setup at the end, by executing the following command:

ssh -l <user> <remotemachine> 'date' 

At the time of verification, you may encounter an "Invalid Port Error" indicating that the SSH setup was not successful.

This can happen if the ssh.exe (sshUserSetupNT.sh script) is not being invoked from the cygwin home directory.

To resolve this issue, ensure the sshUserSetupNT.sh script on the local OMS machine is being executed from within the cygwin (BASH) shell only. The script will fail to execute if done from outside this location.

If there are multiple Cygwin installations, and you want to find out which ssh.exe is being invoked, execute the following command:

C:\Cygwin\bin\which ssh

For example, when you execute the previously mentioned command, and it returns a result that is similar to the following:

\cygdrive\c\WINDOWS\ssh 

This indicates that the ssh.exe file from Cygwin is not being invoked as there is C:\windows that is present before C:\Cygwin\bin in the PATH environment variable.

To resolve this issue, rename this ssh.exe as follows:

-C:\cygwin>move c:\WINDOWS\ssh.exe c:\WINDOWS\ssh.exe1 
          1 file(s) moved.

Now, execute the C:\Cygwin which ssh command again.

The result should be similar to "\usr\bin\ssh".

This verifies that ssh.exe file is being invoked from the correct location (that is, from your C:\Cygwin\bin folder).

HKEY_LOCAL_MACHINE\SOFTWARE\Cygnus Solutions\Cygwin\mounts v2\

sshConnectivity.sh Script Fails

If you are executing the sshConnectivity.sh script on Cygwin version 5.2, the script may fail and result in the following error:

"JAVA.LANG.NOCLASSDEFFOUNDERROR"

To workaround this issue, ensure the Oracle home in the Cygwin style path is defined as follows:

ORACLE_HOME="c:/oraclehomes/oms10g/oracle"

You can find out the currently installed Cygwin version by executing the uname command on the Cygwin window.

Troubleshooting the "command cygrunsrv not found" Error.

During the SSH daemon setup, you may encounter a "command cygrunsrv not found" error. This can occur due to one of the following two reasons:

• The sshd service is not running.

• The Cygwin installation was not successful.

ssh-host-config

cygrunsrv -start sshd

SSH Setup Verification Fails with "Read from socket failed: Connection reset by peer." Error

After the SSH setup is complete, the script automatically executes the following verification command:

ssh -l <user> <remotemachine> 'date'

If this command returns an error stating "Read from socket failed: Connection reset by peer", this means SSH was incorrectly set up. To resolve this issue, go to the remote machine where you attempted to set up user equivalence and do the following:

1. Stop the SSHD service (cygrunsrv -stop sshd).

2. Go to the etc directory (cd /etc).

3. Change the SSH file owner to the appropriate system (chown <SYSTEM> ssh*).

4. Go to the Cygwin command prompt and execute the following:

   chmod 644 /etc/ssh*
   chmod 755 /var/empty
   chmod 644 /var/log/sshd.log
   
   

5. Now, execute the verification command from the Management Service (OMS) machine (ssh -l <user> <remote machine> 'date'). This should display the date correctly, suggesting the SSH setup was successful.

6. Finally, start the SSHD service (from /usr/bin/sshd), or by executing cygrunsrv -start sshd.

7. Now, execute the verification command again from the OMS machine (ssh -l <user> <remote machine> 'date'). This should display the date correctly, suggesting the SSH setup was successful.

SSHD Service Fails to Start

During SSHD configuration, the SSHD service is created for the local account by default. When you log in as a domain user, this account is not recognized by the service, and does not start up.

To resolve this issue, you must change the SSHD service "Log On As" value from LocalSystem to the domain user. To do this, complete the following steps:

1. Right-click on My Computer and select Manage.

2. In the Computer Management dialog box that appears, click Services under Services and Applications.

3. In the right pane, select the Cygwin SSHD service, right-click and go to Properties.

4. In the Cygwin SSHD Properties window that appears, select This Account.

5. Now, specify the appropriate domain name and user (in the form of domain\user, for example, FOO-US\pjohn).

6. Specify the password for this user, and click Apply.

7. Now, go to the Cygwin command prompt and execute the following:

   chmod 644 /etc/ssh*
   chmod 755 /var/empty
   chmod 644 /var/log/sshd.log
   
   

8. Start SSHD by executing the following command:

   /usr/sbin/sshd
   

Timezone Prerequisite Check Fails

The timezone prerequisite check (timezone_check) will fail if the TZ environment variable is not set on the SSH daemon of the remote host.

To resolve this issue, you must set the TZ environment variable on the SSH daemon of the remote host. See AppendixC, "Setting Up the Timezone Variable on Remote Hosts" for more information.

Alternatively, you do the following:

• If you are installing or upgrading the agent from the default software location, set the timezone environment variable by specifying the following in the Additional Parameters section of the Agent Deploy application:

  -z <timezone>
  For example, -z PST8PDT
  
  

• If you are installing the agent from a nondefault software location, you must specify the timezone environment variable using the following command:

  s_timeZone=<timezone>
  For example, s_timezone=PST8PDT
  

OMS Version Is Not Displayed

If the OMS version is not displayed in the log file, it could mean that the installed agent is not registered with a secure and locked Management Service (OMS).

You can verify this by executing the following commands:

emctl status oms
emctl status agent

To resolve this issue, you must manually secure the Management Agent by executing the following command:

<AGENT_HOME>/bin/emctl secure agent -reg_passwd <password>

Discrepancy Between Agent and Repository URL Protocols

If the agent installation is successful, the protocol for both agent and the repository URLs are the same. That is, both URLs start with the https protocol (meaning both are secure).

If the protocol for the agent URL is displayed as http instead of https, this means that the agent is not secure.

To resolve this issue, you must secure the agent manually by executing the following command:

<AGENT_HOME>/bin/emctl secure agent -reg_passwd <password>

Last Successful Upload Does Not Have a Time Stamp

If there is no time stamp against this parameter in the log (displays Null), it means that the agent is unable to upload any data.

To resolve this issue, you must perform a manual upload of the data by executing the following command, and then check the log again:

<AGENT_HOME>/bin emctl upload

emctl status Log File is Empty

If the agent is not ready and running, the emctl status log displays only the copyright information. None of the parameters listed in the sample log is displayed.

The issue can occur due to any of the following reasons:

• Agent is not secure: To manually secure the agent, execute the following command:

  <AGENT_HOME>/bin emctl secure agent -reg_passwd <password>
  
  

• Agent is not running: Check if the agent is running. If not, you can start the agent manually by executing the following command:

  <AGENT_HOME>/bin emctl start agent
  
  

• Agent port is not correct: Verify whether the agent is connecting to the correct port. To verify the port, look into the sysman/config/emd.properties file:

  You must also ensure the following are correctly set in the emd.properties file:

  1. REPOSITORY_URL: Verify this URL (http://<hostname>:port/em/upload). Here, ensure the host name and port are correct.

  2. emdWalletSrcURL: Verify if the host name and port are correct in this URL (http://<hostname>:port/em/wallets/emd).

  3. agentTZRegion: Ensure the time zone that is configured is correct.

Invoking the One-Off Patches Configuration Assistant in Standalone Mode

During the installation process, this configuration assistant is executed before the Management Service Configuration Assistant is run.

This configuration assistant applies the one-off patches that are required for a successful Enterprise Manager 10g Release 2 installation.

To run this configuration assistant in standalone mode, you must execute the following command from the Management Service Oracle home:

<OMS_HOME>/perl/bin/perl <OMS_HOME>/install/oneoffs/applyOneoffs.pl

Invoking the Database Configuration Assistant in Standalone Mode

To run the Database Configuration Assistant, you must invoke the runConfig.sh script as:

<DB_Home>/oui/bin/runConfig.sh ORACLE_HOME=<DB_HOME> ACTION=Configure MODE=Perform

On Microsoft Windows, replace runConfig.sh with runConfig.bat in the previously mentioned command.

Invoking the OMS Configuration Assistant in Standalone Mode

To run the OMSConfig Assistant, you must invoke the runConfig.sh as the following:

<OMS_Home>/oui/bin/runConfig.sh ORACLE_HOME=<OMS_HOME> ACTION=Configure MODE=Perform

On Microsoft Windows, replace runConfig.sh with runConfig.bat in the previously mentioned command.

Invoking the Agent Configuration Assistant in Standalone Mode

To run the AgentConfig Assistant, you must invoke the runConfig.sh as the following:

<Agent_Home>/oui/bin/runConfig.sh ORACLE_HOME=<AGENT_HOME> ACTION=Configure  MODE=Perform

On Microsoft Windows, replace runConfig.sh with runConfig.bat in the above-mentioned command.

Agent_Home/bin/agentca -f

Agent_Home/bin/agentca -f -c "node1,node2,node3,...."

Invoking the OC4J Configuration Assistant in Standalone Mode

If you want to deploy only the Rules Manager, execute the following commands:

/scratch/OracleHomes/oms10g/jdk/bin/java -Xmx512M -DemLocOverride=/scratch/OracleHomes/oms10g -classpath
/scratch/OracleHomes/oms10g/dcm/lib/dcm.jar:/scratch/OracleHomes/oms10g/jlib/e mConfigInstall.jar:/scratch/OracleHomes/oms10g/lib/classes12.zip:/scratch/Orac leHomes/oms10g/lib/dms.jar:/scratch/OracleHomes/oms10g/j2ee/home/oc4j.jar:/scr atch/OracleHomes/oms10g/lib/xschema.jar:/scratch/OracleHomes/oms10g/lib/xmlpar serv2.jar:/scratch/OracleHomes/oms10g/opmn/lib/ons.jar:/scratch/OracleHomes/om s10g/dcm/lib/oc4j_deploy_tools.jar oracle.j2ee.tools.deploy.Oc4jDeploy -oraclehome /scratch/OracleHomes/oms10g -verbose -inifile /scratch/OracleHomes/oms10g/j2ee/deploy.master -redeploy

On Microsoft Windows, replace runConfig.sh with runConfig.bat in the previously mentioned command.

Enterprise Manager Deployment Fails

Enterprise Manager deployment may fail due to the Rules Manager deployment failure.

To resolve this issue, redeploy Enterprise Manager by following these steps:

1. Move OH/j2ee/deploy.master to OH/j2ee/deploy.master.bak.

2. Execute the OH/bin/EMDeploy script.

3. Restore the OH/j2ee/deploy.master. That is, execute mv OH/j2ee/deploy.master.bak OH/j2ee/deploy.master

Oracle Management Service Fails While Deploying Enterprise Manager AgentPush Application

The cfgfw logs display the following error:

Redeploying application 'EMAgentPush' to OC4J instance 'OC4J_EMPROV'. FAILED! ERROR: Caught exception while deploying 'EMAgentPush' to 'OC4J_EMPROV':java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

This error is due to Ipv6 entries in /etc/hosts file. When prompted to execute root.sh or when configuration fails, do the following:

1. In <OMS Home>/sysman/install/EMDeployTool.pm, include "-Djava.net.preferIPv4Stack=true" in the command executed in deployEmEar ().

2. In <OMS Home>/opmn/conf/opmn.xml, include "-Djava.net.preferIPv4Stack=true" in java-options of all OC4J processes.

In 'Enterprise Manager with new Database' Install, Oracle Management Service Configuration Fails While Unlocking Passwords

The cfgfw logs display the following error:

Failed to initialize JDBC Connection

This is caused when listener does not start during NetCA execution and the following error will be present in the installActions log:

Listener start failed. Listener may already be running.

To rectify this error, add the following line in <DB Home>/network/admin/listener.ora:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF

Then, restart the listener.

Dropping of Repository Hangs If SYSMAN Sessions are Active

While installing Enterprise Manager using existing database, Oracle Management Service configuration hangs while dropping the repository. This is due to active SYSMAN sessions connected to the database.

To resolve this issue, shutdown any existing Enterprise Manager sessions (both Grid Control and Database Control) or other SQLPLUS SYSMAN sessions.

If Oracle Management Service Configuration is Retried, oracle.sysman.emSDK.svlt.ConsoleServerHost and oracle.sysman.emSDK.svlt.ConsoleServerName in emoms.properties are Swapped and There is an Extra Underscore in ConsoleServerHost

This problem only occurs with 10.2.0.1.0 Additional Oracle Management Server installation.

To resolve this issue, swap the values and remove the extra underscore in ConsoleServerName in emoms.properties present in <OMS_ORACLE_HOME>/sysman/config directory.

Agent Does Not Start Up After Upgrade

During an agent upgrade from 10.1.0.n to 10.2.0.2, the agent may fail to start up after upgrade if the time zone that is configured for the upgraded agent is different from the originally configured agent.

You can correct this issue by changing the time zone. To do this, execute the following command from the upgraded agent home:

emctl resetTZ agent

This command will correct the agent-side time zone, and specify an additional command to be run against the repository to correct the value there.

Missing Directories When Upgrading Agent from 10.1.0.5 to 10.2.0.1

In a Windows NT RAC agent upgrade scenario, after the AgentOnly shiphome installer has completed installation, the utility <Upgrade AOH>/oui/bin/upgrade has to be executed on every single node in the RAC to complete the agent upgrade.

Steps to Follow for Agent Recovery

Use the following instructions to perform an agent recovery:

1. After exiting the installer, you must open a new window and change the directory to the <New_AgentHome>/bin.

2. Execute the script ./upgrade_recover.

3. You can then start the old agent and continue using it. If you want to remove the installed binaries of the new agent home, use the Remove Productions function of the installer.

Steps to Follow for OMS Recovery

If the schema has been upgraded or the upgrade was incomplete, you must manually restore the database to the backup that was taken prior to executing the OMS upgrade.

You can determine the status of the repository upgrade by looking into the log file at <New_OMSHome>/sysman/log/emrepmgr.log.<proc_id>. The last line of the log file provides the status of the upgrade. If the upgrade was completed without errors, it reads Repository Upgrade Successful. If not, the message Repository Upgrade has errors… is displayed.

Follow these instructions to perform an OMS recovery:

1. Restore the database to the backup. See Oracle Database Administrator's Guide for more information.

2. After the database is restored, start the database and listener to ensure successful restoration.

3. Open a new window and change the directory to the <New_OMSHome>/bin.

4. Now, execute the ./upgrade_recover.

Start the old OMS and continue to use it. If you want to remove the binaries of the newly installed OMS home, use the Remove Productions function in the installer.

Steps to Re-create the Repository

If the Management Service configuration plugin fails due to the repository creation failure, rerunning the configuration tool from Oracle Universal Installer drops the repository and re-creates it. However, if you want to manually drop the repository, complete the following steps:

emctl config emkey -repos -force

java.sql.SQLExecution: ORA-28000: the account is locked during recreation of repository.