B Troubleshooting DCM

B.1.1 OPMN Cannot Start

Problem

During a configuration operation, the opmn.xml file was corrupted or removed. This error may be accompanied by one of the following error messages:

ADMN-202001
An exception has occurred while accessing DCM repository. Please refer to the
 accompanying base exception for more details.
Base Exception:
oracle.ons.ONSException:Unable to open file /private/10g/opmn/conf/opmn.xml

or

ADMN-705002
The most likely causes are: 
1. Configuration error; TaskMaster cannot read configuration file; 
2. Cannot connect to the repository.
Base Exception:
oracle.ons.SubscriptionException:Subscription request timed out after 120000
millseconds. Possible causes: OPMN is not running, you may have OPMN running in an alternate ORACLE_HOME using duplicate port values, OPMN may be misconfigured.

Solution 1

Restart OPMN using the last saved version of the opmn.xml configuration file and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.

To do this, perform the following steps:

1. Perform a opmnctl validate operation on the saved opmn.xml.bak file.

   % cd $ORACLE_HOME/opmn/conf
   % $ORACLE_HOME/opmn/bin/opmnctl validate opmn.xml.bak
   
   

   
   

   Note: The opmn.xml.bak file is created upon installation. It is re-created before any change (through DCM or Oracle Enterprise Manager 10g) is committed.

   
   

2. If step 1 returns the following, then continue to the next step:

   opmnctl: opmn validation succeeded!
   
   

   If step one returns with the message:

   opmnctl: opmn validation failed!
   
   

   Then, if you know how to correct the error in opmn.xml.bak then correct it and try step 1 again. Otherwise, try Solution 2.

3. Replace the opmn.xml file with the file that was saved in the last DCM operation. The saved version of opmn.xml should be in the file $ORACLE_HOME/opmn/conf/opmn.xml.bak.

   For example:

   % cd $ORACLE_HOME/opmn/conf
   % cp opmn.xml.bak opmn.xml
   
   

4. Start the DCM shell and run the resyncinstance command.

   % cd $ORACLE_HOME/dcm/bin
   % dcmctl shell
   dcmctl> resyncinstance -force
   All OC4J components are stopped during a resyncinstance -force.
   They will need to be restarted after the command completes.
   Current State for Instance:10gM21.tvanraal-sun.us.oracle.com
   
   

5. Use opmnctl to check the status of all components, and if any components that you expect to be running are not running, then start them:

   % cd $ORACLE_HOME/opmn/bin
   % opmnctl status
   

Solution 2

Modify the last saved version of the opmn.xml file to allow DCM to run, and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.

To do this, perform the following steps:

1. Perform a opmnctl validate operation on the saved opmn.xml.bak file.

   % cd $ORACLE_HOME/opmn/conf
   % $ORACLE_HOME/opmn/bin/opmnctl validate opmn.xml.bak
   
   

2. If step one returns the following:

   opmnctl: opmn validation failed!
   
   

   Then, you need to correct the error in opmn.xml.bak to allow you to start the dcm daemon. First, backup your current version of opmn.xml.bak, in case you have problems, so you can revert to your backup copy. For example, use the following command:

   cp opmn.xml.bak opmn.xml.bak.orig
   
   

   Edit the $ORACLE_HOME/opmn/conf/opmn.xml.bak file and add the attribute status="disabled" for each <ias-component> element in opmn.xml.bak. For only the component dcm-daemon, set the status= "enabled".

   For example:

   <ias-component id="OC4J" status="disabled">
   <ias-component id="dcm-daemon" status="enabled" id-matching="true">
   
   

3. Replace the opmn.xml file with the updated version of opmn.xml.bak.

   For example:

   % cd $ORACLE_HOME/opmn/conf
   % cp opmn.xml.bak opmn.xml
   
   

4. Start the DCM shell and run the resyncinstance command.

   % cd $ORACLE_HOME/dcm/bin
   % dcmctl shell
   dcmctl> resyncinstance -force
   All OC4J components are stopped during a resyncinstance -force.
   They will need to be restarted after the command completes.
   Current State for Instance:10gM21.tvanraal-sun.us.oracle.com
   
   

5. Use opmnctl to check the status of all components; if any components that you expect to be running are not running, start them:

   % cd $ORACLE_HOME/opmn/bin
   % opmnctl status
   

Solution 3

If Solution 1 and Solution 2 do not solve this problem, then both the opmn.xml and the opmn.xml.bak files have been corrupted. In this case, you need to manually create an opmn.xml file that enables you to start the dcm-daemon and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.

To create a version of opmn.xml that is sufficient for starting the dcm-daemon, perform the following steps:

1. Copy and paste the text shown in Example B-1 to $ORACLE_HOME/opmn/conf/opmn.xml.

2. Edit the port element such that the attributes local, remote, and request have the correct values for your environment. These values are shown as XXXX, YYYY, and ZZZZ in Example B-1.

3. Using this file, the OPMN Server should start up, which should allow the DCM Daemon to start so that you can restore the opmn.xml file from the DCM repository using the dcmctl resyncInstance command.

<opmn xmlns="http://www.oracle.com/ias-instance">
  <notification-server>
    <port local="XXXX" remote="YYYY" request="ZZZZ"/>
    <log-file path="$ORACLE_HOME/opmn/logs/ons.log" level="4"
      rotation-size="1500000"/>
    <ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
  </notification-server>
  <process-manager>
    <log-file path="$ORACLE_HOME/opmn/logs/ipm.log" level="4"
      rotation-size="1500000"/>
    <process-modules>
      <module path="$ORACLE_HOME/opmn/lib/libopmndcmdaemon">
        <module-id id="DCMDaemon"/>
      </module>
    </process-modules>
    <ias-instance id="foo">
      <environment>
        <variable id="TMP" value="/tmp"/>
      </environment>
      <ias-component id="dcm-daemon" status="enabled" id-matching="true">
        <process-type id="dcm-daemon" module-id="DCMDaemon">
          <process-set id="dcm" numprocs="1">
            <module-data>
              <category id="start-parameters">
                <data id="java-parameters"
                  value="-Xmx256m -Doracle.ias.sysmgmt.logging.loglevel=ERROR -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=$TMP" />
                <data id="jar-file" value="$ORACLE_HOME/dcm/lib/dcm.jar"/>
                <data id="application-parameters" value="daemon -logdir $ORACLE_HOME/dcm/logs/daemon_logs 
 -o $ORACLE_HOME" />              </category>
            </module-data>
          </process-set>
        </process-type>
      </ias-component>
    </ias-instance>
  </process-manager>
</opmn>

B.1.2 DCM Daemon Cannot Start

Problem

The DCM daemon does not start.

Solution

This can occur due to insufficient memory problem or an OPMN problem. You can investigate this problem as follows:

1. Review the error messages in ORACLE_HOME/opmn/logs/dcm-daemon~dcm-daemon~dcm~1.

2. Review the error messages in the log files in ORACLE_HOME/opmn/logs/*.

3. Ensure that the ports OPMN uses are free of conflicts. These ports are listed in ORACLE_HOME/opmn/conf/opmn.xml under the <notification-server> element.

B.1.3 DCM Daemon Does Not Start As Expected

Problem

After a successful execution of the opmnctl startall command, the opmnctl status shows that dcm-daemon is down.

Solution

Start the dcm-daemon component explictly, if necessary, by issuing this command:

opmnctl startproc ias-component=dcm-daemon

By default, the dcm-daemon component is not started by the opmnctl startall command. Instead, the dcm-daemon component is started during initialization by dcmctl or Oracle Enterprise Manager 10g Application Server Control Console.

Starting the dcm-daemon explicitly is required in some cases. For example, if you are performing a cluster-wide DCM operation, such as deploying an application to the cluster, then the dcm-daemon must be running on all of the Oracle Application Server Instances in the cluster. A dcmctl command on the local Oracle Application Server Instance will start the dcm-daemon on the local Oracle Application Server Instance, but not in remote Oracle Application Server Instances of the cluster.

B.1.4 DCM Operations Fail

Problem

Occasionally, while using dcmctl commands with a file-based repository, unexpected messages or failures may occur.

Solution 1

Purge outdated records from the repository data tables using the fileCleanUp command.

Solution 2

Restart all dcmctl processes, dcm daemons, and Application Server Control Console on all the Oracle Application Server Instances in the farm. Restarting these processes may fix the problem, and will not result in a loss of configuration information.

B.1.5 Invalid username/password

Problem

dcmctl returns "invalid username/password; logon denied"

Solution

This could mean that DCM is having problems accessing Oracle Internet Directory or the DCM schema in the metadata repository. Refer to "Troubleshooting Oracle Internet Directory" for the steps for troubleshooting this problem.

B.1.6 BOM (Bill of Materials) Not Updated

Problem

The DCM log file contains the entry "BOM Not Updated". Usually, this error is accompanied by a detailed message resembling the following:

Error writing file path to file for plugin plugin name.

Solution

This message is logged when configuration files for the Oracle Application Server Instance are not synchronized with the configuration information stored in the DCM repository. DCM was not able to update the configuration files due to a file system full condition, file permissions problem, or an operating system problem. Examine the following log files to find the reasons for which DCM could not write to the file system:

ORACLE_HOME/dcm/logs/dcmctl_logs/log.xml

ORACLE_HOME/dcm/logs/emd_logs/log.xml

ORACLE_HOME/j2ee/home/log/log.xml

Another way to pinpoint a problem writing to the file system is to log in to the operating system as the user that installed Oracle Application Server, and attempt to write to a configuration file. The operating system will return an error message that describes the problem precisely.

After you resolve the file system problem, you can resynchronize the Oracle Application Server Instance with this command:

ORACLE_HOME/dcm/bin/dcmctl resyncinstance

B.1.7 In Sync Status is False

Problem 1

dcmctl getState displays an In Sync status of "False: Configuration" for one or more components.

Solution 1

This indicates that the configuration files for the Oracle Application Server Instance are out of sync with the configuration information stored in the DCM repository. This can happen if a previous operation was not able to complete properly. To resolve this problem, you need to restore the configuration information from the DCM repository to the Oracle Application Server Instance as follows:

1. Stop Oracle Enterprise Manager 10g Application Server Control Console to ensure that no attempts are made to change the configuration while you are synchronizing it.

2. Resynchronize the Oracle Application Server Instance with this command:

   ORACLE_HOME/dcm/bin/dcmctl resyncinstance

   This will usually fix the problem. If not, the resyncinstance operation will return error information that will help you determine what is wrong and how to proceed.

Problem 2

dcmctl updateconfig returns the message "The configuration files for this Oracle Application Server Instance are inconsistent with the configuration stored in the repository."

Solution 2

This indicates that the configuration files for the Oracle Application Server Instance are out of sync with the configuration information stored in the DCM repository. Stop and resynchronize the Oracle Application Server Instance. After the configuration files for the Oracle Application Server Instance are synchronized, you can make your original modifications. Be sure to run dcmctl updateconfig again after making the changes.

Problem 3

A DCM configuration file has zero bytes after an operation.

Solution 3

A file size of zero bytes indicates there was a problem writing to the configuration file during a DCM operation. To resolve the problem:

1. Log on to the operating system as the user that installed the Oracle Application Server Instance, and:

   1. Check the file permissions.

   2. Attempt to write to the configuration file, noting the operating system error message after any failure. Resolve the condition that caused the failure.

   3. Verify file system quotas.

   4. Ensure that there are no problems with the operating system that are preventing writing to the configuration file.

2. Resolve any problems found.

3. Restore the file from the DCM repository with the resyncInstance command.

B.1.8 Unable to Connect to the Directory

Problem

dcmctl returns ADMN-100999 and the base exception is "Unable to connect to Directory."

Solution

This error occurs when DCM cannot access Oracle Internet Directory. DCM needs to connect to the directory in order to retrieve the passwords it uses to connect to the Database-based repository.

See if restarting Oracle Internet Directory solves the problem. If it does not, refer to "Troubleshooting Oracle Internet Directory".

B.1.9 Cannot Access the Infrastructure Database

Problem

dcmctl returns ADMN-202026 and the base exception is "Unable to connect to Directory."

Solution

This error occurs when DCM cannot access Oracle Internet Directory. DCM needs to connect to the directory in order to retrieve the passwords it uses to connect to the Database-based repository.

See if restarting Oracle Internet Directory solves the problem. If it does not, refer to "Troubleshooting Oracle Internet Directory".

B.1.10 The dcm.conf File is Unreadable

Problem

An error occurs when attempting login to Application Server Control Console or invoking a dcmctl command; permission is denied when attempting to read the dcm.conf file. This error occurs when a user other than the user who installed the Oracle Application Server Instance invokes a dcmctl command, or when the file permissions on the dcm.conf file have been altered from the default set by the installer.

Solution

Change the ownership/permissions on the ORACLE_HOME/dcm/config/dcm.conf file to the permissions set by the installer when the Oracle Application Server Instance was installed. In addition, ensure that the permissions are set correctly on ORACLE_HOME/opmn/conf/ons.conf and all files under ORACLE_HOME/dcm/logs/dcmctl_logs.

B.1.11 The dcm.conf or dcmCache.xml File is Empty

Problem

Occasionally, during configuration management operations, configuration files in are written out as zero length files. This is usually due to disk write problems, such as lack of space on the disk.

Solution

Copy the corresponding .bak file, dcm.conf.bak or dcmCache.xml.bak, to dcm.conf or dcmCache.xml respectively, and start the Oracle Application Server Instance.

B.1.12 dcmctl getState Hangs

Problem

When the dcmctl getState -d command is executed, it returns the following and then stops:

Current state for Instance:myinstance.example.com

Component Type Up Status In Sync Status

=============================

Solution

Determine which process has actually stopped. It could be that a process associated with one of the related components (that is, OHS, OPMN, OC4J) has stopped.

1. Determine the PID of the dcm-daemon process, using the following command:

   ORACLE_HOME/opmn/bin/opmnctl status

2. Send a kill -3 signal to the dcm-daemon process with the following command:

   kill -3 PID

   This tells the JVM to dump an execution stack trace.

3. Open the ORACLE_HOME/opmn/logs/dcm-daemon~dcm-daemon~dcm~1 file and examine the execution stack trace to identify the process that caused DCM to stop.

B.1.13 getState Cannot Be Used With (9.0.2 or 9.0.3) Instances

Problem

Invoking the command dcmctl getstate -i instance name from an Oracle Application Server 10g Release 2 (10.1.2) Oracle home results in an ADMN-604104 exception. This problem occurs when the instance name is a 9.0.2 or 9.0.3 Oracle Application Server Instance.

Solution

When trying to retrieve the state of a 9.0.2 or 9.0.3 Oracle Application Server Instance, invoke the dcmctl utility from a 9.0.2 or 9.0.3 Oracle home, and not from the Oracle Application Server 10g Release 2 (10.1.2) Oracle home.

B.1.14 dcmctl joinFarm Produces Error

Problem

When the dcmctl joinfarm command is executed, it returns the following:

ADMN-202026
A problem has occurred accessing the OracleAS 10g infrastructure database.
Base Exception:
oracle.ias.repository.schema.SchemaException:DCM Repository is not Configured
Please refer to the base exception for resolution or call Oracle support.
oracle.ias.repository.schema.SchemaException: DCM Repository is not Configured
.
.
.

Solution

If you are attempting to join a File-based repository, issue the command again, using the -r option to specify the repository ID for the File-based repository.

The -r option is required to specify the ID of a File-based repository. If you erroneously omit the -r option, dcmctl attempts to join an Infrastructure database farm with the repository ID you provided.

B.1.15 repositoryRelocated Command Causes Repository to be Unreachable

Problem

You executed repositoryRelocated and now the repository is unreachable.

The repositoryRelocated command is intended to be used in conjunction with the importRepository command. A Oracle Application Server Farm can only be associated with one repository at a time. When the repository is imported to a new Oracle Application Server Instance of the Oracle Application Server Farm, the old Oracle Application Server Instance must be notified, with the repositoryRelocated command, that it is no longer the repository host. If the repository was not imported to a new Oracle Application Server Instance, then there is currently no Oracle Application Server Instance managing the repository.

Solution

The solution depends on the circumstances under which the command was issued.

If the intent was to import an exported repository image to a new Oracle Application Server Instance, simply issue the importRepository command. After the repository is imported to a new Oracle Application Server Instance, all Oracle Application Server Instances will be able to locate the new repository host. If the command was run in error, and you wish to continue to use the current repository:

1. Stop the Oracle Application Server Instance.

2. Navigate to the ORACLE_HOME/dcm/config directory.

3. Compare the files with the command:

   diff dcm.conf dcm.conf.bak

   The result should be:

   < repository_type=cache

   > repository_type=cache rep

4. Restore the dcm.conf file to the original version with the command:

   cp dcm.conf.bak dcm.conf

5. Start the Oracle Application Server Instance.

   The repository is now available to the farm.

B.1.16 Unreachable IP Address

Problem

The following Application Server Control Console or dcmctl errors may occur if the configuration file ORACLE_HOME/dcm/config/dcmCache.xml includes a <communication> element with a <coordinator> subelement that specifies an ip attribute for an IP address that is not reachable.

The exception 202504, has occurred in the cache layer of the persistence manager. "Network port is in use.". Resolution: Please, refer to the base exception for the details.

And the following dcmctl error is also associated with this problem:

Local Stack:
oracle.ias.cache.group.GroupException: Network port is in use
at oracle.ias.cache.group.Transport.ssinit(Unknown Source)
at oracle.ias.cache.group.Transport.init(Unknown Source)

Solution

The IP address could become unreachable if the IP address for the machine changes, for example, in a DHCP environment.

Use the following procedure to workaround this problem:

1. Change the ip attribute specified in the <coordinator> element of the ORACLE_HOME/dcm/config/dcmCache.xml configuration file. There are several ip attributes and <coordinator> elements specified in this file; change the <coordinator> element with attribute "original=true". In this element change the ip attribute to the correct value for the machine. Use the hostname and nlslookup commands or similar commands to identify the correct IP address.

2. Change the VirtualHostName value in ORACLE_ HOME/config/ias.properties to the corresponding correct hostname.

3. Run the command dcmctl resetHostInformation.

This should correct the problem.

B.1.17 resyncInstance Command Does Not Restore Properly

Problem

A configuration change was not reverted as expected in the following sequence of steps:

1. Update of DCM-managed configuration file

2. Execute the updateConfig command

3. Manually remove change to configuration file

4. Execute the resyncinstance command

Solution

Make DCM aware of the removal of the configuration change in Step 3 by issuing the updateConfig command after Step 3. You can use the -force option to force DCM to resynchronize the Oracle Application Server Instance whether it thinks it necessary or not:

ORACLE_HOME/dcm/bin/dcmctl resyncinstance -force

B.2.1 Troubleshooting Oracle Process Manager and Notification Server

A problem with starting or stopping could be due to OPMN, or to one of the components it is attempting to start and stop. The OPMN logs are the starting point for determining the source of the problem. The OPMN logs are located in the ORACLE_HOME/opmn/logs/. In addition to the main logs, opmn.log, ipm.log, and ons.log, there are log files for each component managed by OPMN.

B.2.2 Troubleshooting Oracle Internet Directory

If DCM returns an error referring to the directory, or a problem with a password, problems with Oracle Internet Directory may be the cause. This section provides instructions on how to resolve these types of problems.

B.2.3 Troubleshooting the Metadata Repository

If DCM is reporting problems with the repository, and you are using the metadata repository (not a File-based repository), you can troubleshoot the repository as described in this section.

B.2.4 Troubleshooting the File-based Repository

If DCM is reporting problems with a File-based repository, you can troubleshoot the repository as described in this section.