J Troubleshooting Oracle Internet Directory

J.1.2.1 Do Not Use TCP-Based Monitoring of Oracle Internet Directory Server Availability on Microsoft Windows 2003 Server

If you use the F5 load balancer for monitoring Oracle Internet Directory server availability, configure the load balancer to use LDAP- or HTTP-based monitoring, as described in the Oracle Application Server High Availability Guide section "Configuring A Load Balancer For OracleAS Cluster (Identity Management)." Using TCP-based monitoring might cause the service to become unavailable, due to an operating system bug on Windows 2003 Server.

J.1.2.2 Do Not Install DaimondCS Port Explorer

Oracle Internet Directory will not work if DaimondCS Port Explorer is installed on the system.

J.1.3.1 Oracle Database Server Error Due to Schema Modifications

You get error ORA-1562

Problem

If you attempt to add more schema components than can fit in the rollback segment space, you will encounter this error and the modifications will not commit.

Solution

To solve this, increase the size of the rollback segments in the database server.

J.1.3.2 Constraint Violation Error Due to Editing a User or Group or Creating a Realm

You get the following error in oidldap*.log:

ORA-01483: invalid length for DATE or NUMBER bind variable.

You may also see the following error on your screen:

LDAP: error code 19 - Constraint Violation

These errors might only occur intermittently.

Problem

If you loaded the OracleAS Metadata Repository into an Oracle 10g Database that uses the AL32UTF8 character set, you may encounter some errors when you try to edit a user or Group, or Create Identity Management Realms in Oracle Internet Directory. Editing a user includes editing attributes for an existing user.

Solution

As a workaround, you can wait a bit and try editing the user again.

J.1.3.3 Standard Error Messages Returned from Oracle Directory Server

Table J-1 lists standard error messages and their causes. Oracle Internet Directory also returns other messages listed and described in "Additional Directory Server Error Messages".

J.1.3.4 Additional Directory Server Error Messages

Table J-2 lists additional directory server error messages and their causes. These messages do not display error codes.

The Oracle Internet Directory application replaces the parameter tag seen in some of the following messages with the appropriate runtime value.

J.1.4.1 Password Policy Error Messages

Table J-3 contains the error messages sent to the client as a result of password policy violations. The error codes are not standard LDAP error codes. They are messages sent as a part of additional information in the LDAP result.

J.1.4.2 Possible Password Policy Problems

This section describes some of the potential problems with password policies and the corresponding solutions.

J.1.5.1 Poor LDAP Search Performance

LDAP search performance is poor.

Problem

Various problems.

Solution

Make sure that:

• Schema associated with the ODS user is ANALYZED

• For searches involving multiple filter operands, make sure that the order in which they are given goes from the most specific to the least specific. For example, &(uid=john.doe)(objectclass=person) is better than &(objectclass=person)(uid=john.doe).

J.1.5.2 Poor LDAP Add or Modify Performance

LDAP add or modify performance is poor.

Problem

Various problems

Solution

Make sure that:

• There are enough redo log files in the database

• The undo tablespace in the database is large enough

• The schema associated with the ODS user is ANALYZED

When estimating the statistics, you can use the OID Database Statistics Collection tool to analyze the various database ODS schema objects.

Both the tracing functionality described in "Using Debug Logging" and the database tracing event 10046 can assist you in diagnosing performance issues.

J.1.6.1 About the Tools for Starting, Stopping, and Restarting the Directory Server Instance

The tool usedto start and stop the directory server as an Oracle Application Server component is OPMN. For information on troubleshooting OPMN, see the "Troubleshooting" appendix in Oracle Process Manager and Notification Server Administrator's Guide.

There are two tools used to start, stop, and restart directory server instances: OID Control Utility (OIDCTL) and OID Monitor (OIDMON).

OIDCTL When OIDCTL is executed, it connects to the database as user ODS. Depending on the options used in the command, it either inserts or updates rows into a table named ODS.ODS_PROCESS. If the START option is used, then a row is inserted. If either the STOP or RESTART option is used, then a row is updated.

The ODS.ODS_PROCESS table includes the following information:

• instance—The unique number of the instance, any value between 0 and 1000

• pid—Process identifier, which will be updated by OIDMON when the process is started

• state—The type of operation requested

  The possible values for state are:

  • 0=stop

  • 1=start

  • 2=running

  • 3=restart

  • 4=shutdown

  • 5=failedover

OIDMON To start, stop, or restart a directory server instance, OIDMON must be running. At specified intervals, this daemon checks the value of the state column in the ODS.ODS_PROCESS table.

If it finds a row with state=0, then it reads the pid and stops the process.

If it finds one with state=1 or state=4, then it starts a new process and updates the pid column with a new process identifier.

If it finds one with state=2, then it reads the pid and verifies that the process with that pid is running. If it is not running, then OIDMON starts a new process and updates the pid column with a new process identifier.

If it finds a row with state=3, then OIDMON reads the pid, stops the process, starts a new one, and updates the pid accordingly.

If OIDMON cannot start the server for some reason, it retries. If it is not running on a node in an Oracle Application Server Cluster (Identity Management) configuration, and it is still unsuccessful after 10 retries, it deletes the row from the ODS.ODS_PROCESS table. If OIDMON is running on a node in a Oracle Application Server Cluster (Identity Management) configuration, it retries 100 times.

If it is still unsuccessful, it pushes the request to another node.

In short, OIDCTL inserts and updates state information in the rows in the ODS.ODS_PROCESS table. OIDMON then reads that information and performs the specified task.

About the Processes Involved in Starting, Stopping, and Restarting the Directory Server

Starting, stopping and restarting the directory server involves a number of processes. OIDMON is one process. On Unix, it is called oidmon. In a Microsoft Windows environment, it is called oidmon.exe.

To start an instance, OIDMON checks the unique number in the instance column mentioned in the previous section. It then starts another process, namely, the listener/dispatcher, which is different from the Oracle Net Services listener process. It stores the process identifier for that new process in the pid column.

The listener/dispatcher, in turn, starts a number of server processes as defined in the configuration set entry. Note that these server processes are controlled by the listener/dispatcher and not by OIDMON. If one of these processes fails, then it is automatically restarted by the listener/dispatcher.

Together, the listener/dispatcher and the server processes constitute a directory server instance. On UNIX, this directory server instance is called oidldapd. On Microsoft Windows, they are called oidldapd.exe.

In short, there are at least three processes: one for OIDMON and at least two for the directory server itself. When all processes are running, you should see something like the following on UNIX computers:

% ps -ef|grep oid
root 12387 12381 0 Mar 28 ? 0:05 oidldapd -i 1 -conf 0 key=811436710
root 12381 1 0 Mar 28 ? 0:10 oidmon start
root 13297 1 0 Mar 28 ? 0:14 oidldapd

Another way to obtain server information is by running ldapcheck. When you do this, you may see something like this:

Checking Oracle Internet Directory Processes ...
Process oidmon is Alive as PID 12381
Process oidldapd is Alive as PID 12387 
Process oidldapd is Alive as PID 13297
Not Running ---- Process oidrepld

J.1.6.2 Problems Starting, Stopping, and Restarting the Directory Server

This section describes some problems you might have when starting, stopping, or restarting the directory server.

-rws--x---    1 root     dba       1691802 Jan 20 10:30 oidldapd

cd $ORACLE_HOME/bin
chown root:dba oidldapd
chmod 0710 oidldapd    
chmod u+s oidldapd

netstat -a | grep portNum

J.1.7.1 Replication Server Does Not Start

There are several problems that can prevent the replication server from starting.

Problem

Invalid oidctl syntax

Solution

Use the following syntax to start the replication server.:

oidctl server=oidrepld connect=connect string instance=instance_number \
       flags="-h host -p port"

Problem

Oracle Internet Directory is not running at the host and port you specified on the command line when you attempted to start the replication server. This caused the anonymous bind to the target Oracle Internet Directory to fail.

Solution

Make sure the target Oracle Internet Directory is up and running at the specified host and port.

Problem

The replication server is attempting to bind to the host and port specified in either the orclreplicaprimaryurl or the orclreplicasecondaryurl attribute of the Replica entry, but Oracle Internet Directory is running at a different host or port.

Solution

If you decide to run Oracle Internet Directory at a different host or port, add the new information to the orclreplicasecondaryurl attribute of the replica entry, as follows:

1. Prepare a modification file, mod.ldif. For example, to change to host my.us.oracle.com and port 4444, you would specify:

   dn: orclreplicaid=replica_ID, cn=replication configuration
   changetype: modify 
   add: orclreplicasecondaryurl 
   orclreplicasecondaryurl: ldap://my.us.oracle.com:4444/ 
   
   

2. Run:

   ldapmodify -h host -p port -f mod.ldif
   
   

Problem

The ReplBind credential in the replication wallet $ORACLE_HOME/ldap/admin/oidrORACLE_SID is corrupt or invalid. That is, the password stored in the wallet is not the same as the password that is stored in the directory, or the wallet does not exist. This causes the replication bind to fail and the replication server to exit with an error.

You might see messages similar to this example in the file oidrepldXX.log:

2005/07/21:11:13:28 * gslrcfdReadReplDnPswd:Error reading repl passwd 
2005/07/21:11:13:28 * gslrcfcReadReplConfig:Error found. 
2005/07/21:11:13:28 * Failed to read replication configuration information. 

Solution

Use remtool to fix the replication bind credential in the replication wallet or to synchronize between Oracle Internet Directory and the replication wallet.

• remtool -pchgpwd changes the password of the replication dn of a replica. Use this option if you know the current replication DN password stored in the directory and you want to change it both in the directory and in the wallet.

• remtool -presetpwd resets the password or the replication dn of a replica. Use this option if you know the current replication DN password stored in the directory and you want to change it both in the directory and in the wallet.

• remtool -pchgwalpwd changes password of replication dn of a replica only in the wallet. Use this option if you know the replication DN password stored in the directory but you are not sure whether the wallet has the correct password or you want to create the wallet file.

All of these options will create a wallet if one does not already exist.

J.1.7.2 Repository Creation Assistant Error

Problem

When you use the Oracle Application Server tool RepCA to load Oracle Internet Directory schema into an existing Oracle 10.1.0.3 Database, you might see the following error message in the $ORACLE_HOME/assistants/repca/log/repca*log file:

SP2-0332: Cannot create spool file.

Solution

This error message can be ignored.

J.1.7.3 Errors in Replication Bootstrap

A number of errors can occur in replication bootstrap.

Problem

Some of the naming contexts failed to be bootstrapped.

Solution

Identify the naming contexts that failed to be bootstrapped, and use the oidreconcile tool to reconcile them.Then resume replication by setting the consumer's replica state to ONLINE mode

Problem

Various causes.

Solution

Identify the cause of the bootstrap failure and fix the cause, then restart bootstrapping by setting consumer's replica state to BOOTSTRAP mode.

Solution

To determine the exact cause of the error, examine the log file oidldapdxx.log. Look for error messages like those in the following example:

2004/09/14:12:57:23 * Starting OIDREPLD against dlsun1418:4444...
2004/09/14:12:57:25 * Starting scheduler...
2004/09/14:12:57:26 * Start to BootStrap from supplier=dlsun1418_replica to consumer=dlsun1418_replica2
2004/09/14:12:57:27 * gslrbssSyncDIT:Replicating namingcontext=cn=oraclecontext ......
2004/09/14:12:58:21 * gslrbssSyncDIT:Sync done successfully for namingctx: cn=oraclecontext, 222 entries matched
2004/09/14:12:58:21 * gslrbssSyncDIT:Replicating namingcontext=cn=joe smith ......
2004/09/14:12:58:23 * BootStrap failure when adding DN=cn=Joe Smith,
server=dlsun1418_replica2,err=Constraint violation.
2004/09/14:12:58:23 * gslrbssSyncDIT:Sync failed for namingctx: cn=joe smith, only 1 entries retrieved
2004/09/14:12:58:23 * gslrbssSyncDIT:Replicating namingcontext=cn=oracleschemaversion ......
2004/09/14:12:58:25 * gslrbssSyncDIT:Sync done successfully for namingctx: cn=oracleschemaversion, 10 entries matched
2004/09/14:12:58:51 * gslrbsbBootStrap: Failure occured when bootstrapping 1 out of 3 namingcontext(s) from the supplier

Identify the cause of the bootstrap failure and fix it. You can identify the naming contexts that caused the problem, then use oidreconcile to compare and reconcile the naming contexts. Once you have resolved the problem, start bootstrapping again by starting the Oracle Internet Directory replication server.

Problem

The Oracle Internet Directory server was shut down during the bootstrapping

Solution

Make sure both the supplier Oracle Internet Directory and the consumer Oracle Internet Directory servers are up and running during replication bootstrapping.

Problem

Some of the entries being bootstrapped cannot be applied at the consumer due to a constraint violation.

Solution

Make sure the Oracle Internet Directory schema of the consumer are synchronized with those of the supplier before starting replication bootstrap. When you add an LDAP replica, remtool ensures that the Oracle Internet Directory schema on the consumer replica are synchronized with those on the supplier replica.

Problem

Improper replication filtering during bootstrapping. Replication supports excluding one or more attributes during bootstrapping. However, if a mandatory attribute of an entry is configured to be excluded, that entry cannot be applied at the consumer due to an objectclass violation.

Solution

Follow the replication naming context configuration rules in Chapter 25, "Oracle Internet Directory Replication Administration" to configure replication filtering properly.

If you are debugging LDAP replication, you should become familiar with the LDAP replica states. If LDAP-based replication is configured, when the replication server starts, it reads the replica state from the local replica. The replication server behaves differently, depending upon the local replica state. LDAP replication errors appear in oidldapdxx.log

Problem

When you restart the replication server after the replication server failed to bootstrap a naming context having more than 5000 entries, you may see error messages similar to this in the log file oidrepld00.log:

2005/04/05:13:21:55 * gslrbssSyncDIT:Replicating namingcontext=dc=com ...... 
2005/04/05:15:36:09 * gslrbssSyncDIT:Subtree delete on dc=com failed. 
Error=DSA is unwilling to perform 
2005/04/05:15:36:09 * gslrbssSyncDIT:Sync failed for namingctx: dc=com, only 
0 entries retrieved 

The replication server performs two steps during bootstrap operation. First, in the consumer, it deletes the naming contexts that it has to bootstrap. Second, it copies entries belonging to those naming contexts from supplier to consumer. Deletion by the replication server of a naming context having several thousands of entries results in a big transaction. The undo tablespace needs to have sufficient space to accomodate a big transaction. If the database's undo tablespace does not have sufficient space, it will result in an ORA-30036 error.

Solution

Either have the database administrator add more space to the undo tablespace, or use the bulkdelete tool to delete the required naming context before you start the replication server.

J.1.7.4 Changes Are Not Replicated

Changes are not replicated from one node to another.

Problem

The replication server has run out of table space

Solution

Look for the following message in the server log:

OCI Error ORA-1653 : ORA-01653: unable to extend table ODS.ASR_CHG_LOG by 8192 in tablespace OLTS_DEFAULT

Extend the table space and investigate why the table space keeps growing.

Problem

The target Oracle Internet Directory server is down.

Solution

Restart the target Oracle Internet Directory server.

Problem

Various causes

Solution

Make sure the replication server is started on all nodes, in multi-master replication, and at the consumer node in single-master or fan-out replication.

For multi-master Oracle Database Advanced Replication, use remtool to diagnostic and fix problems.

• remtool -asrverify verifies the correctness of a DRG setup and reports problems.

• remtool -asrrectify verifies the correctness of a DRG setup, reports problems, and attempts to rectify the problems.

Check the replication log and LDAP log for error messages and fix the cause of the error after investigation.

J.1.7.5 Replication Stops Working

Problem

Data is not replicated between the replicas. In some cases, a working replication setup stops working after OID Human Intervention Queue entries are applied to one of the nodes. In other cases, adding or deleting a new replica causes problems or failures.

Problem

Various causes

Solution

See the following Oracle MetaLink notes on Oracle MetaLink, http://metalink.oracle.com:

Note 171693.1, "Resolving Conflicts"

Note 122039.1, "Troubleshooting Basics for Advanced Replication"

Note 213910.1, "Debugging OID Replication when ASR_CHG_LOG Never Gets Populated."

You can search for Oracle MetaLink notes by entering a term such as "replication" into the search box.

J.1.9.1 Change Logs Are Not Purged

Change logs grow very large.

Problem

Change logs are not being purged due to a replication issue. For example, if a replication server has been down for a few days, replication change logs will not be purged because they are needed for replication recovery.

Solution

Resolve the replication issue. See "Troubleshooting Oracle Internet Directory Replication"".

Problem

The attribute orclpurgetargetage is set too high and there are one or more enabled but inactive change log subscribers that do not update orclLastAppliedChangeNumber in their subscriber profiles. Change number-based purging won't purge change logs that are not yet consumed and time-based purging won't purge them because they're not old enough.

Solution

Set the attribute orclpurgetargetageto a smaller value so that change logs are purged sooner.

Solution

Disable inactive changelog subscribers so that change logs are purged by change log number-based purging. Locate such enabled but incactive subscriber profiles by examining the orclLastAppliedChangeNumber in all subscriber profiles by typing:

ldapsearch -v -p port -h host -D cn=orcladmin -w password \
           -b "cn=changelog subscriber,cn=oracle internet directory" \
           -s sub "objectclass=orclchangesubscriber" \
           orcllastappliedchangenumber orclsubscriberdisable
 

Look for an entry that has orclSubscriberDisabled equal to zero and an orclLastAppliedChangeNumber value that never changes. If such an entry exists, and the change log garbage collector's orclpurgetargetage is zero or greater, delete the value of orclpurgetargetage. When orclpurgetargetage is not defined or less than zero, the garbage collector will purge changes applied by the replication server, even if another subscriber has not updated its orclLastAppliedChangeNumber.

J.1.11.1 Oracle Internet Directory Server Does Not Start

Either oidctl or opmn fails to start an Oracle Internet Directory server instance.

Problem

The password stored in the oidpwdlldap1 wallet is not synchronized with the ODS password in the backend database.

Solution

Try to connect to the database again using the sqlplus command:

sqlplus ods /ods_password@connect_string

If the connection succeeds, try to synchronize the password in the wallet with the ODS password by using the oidpasswd tool to create a new wallet with the correct password. For example:

>> oidpasswd connect=connect_string create_wallet=true

If the connection attempt fails, you must login into the backend database as a database administrator and change the ODS password by using the sql command:

>> alter user ods identified by some_new_password

Then try to create a new oidpwdlldap1 to store the new password.

Solution

Try to start the Oracle Internet Directory server again.

The oidpwdrSID file contains the DN and password of a replica DN in an encrypted format. The Oracle Internet Directory replication server uses the credential to connect to the Oracle Internet Directory server at startup time.

This is an example of a replication password wallet, oidpwdrSID:

/------BEGIN REPL CREDENTIAL:cn=replication dn,orclreplicaid=qdinh-sun_
adeldap,cn=replication configuration-----
ezNkZXMtY2JjLXBrY3M1cGFkfQUnaz0TsfzcP0nM1HcHAXchf5mJw+sb4y0bLvvw3RvSg7H
S7/WsKJB02fdSGRlmfWAV+6llkRQ26g==
-----END REPL CREDENTIAL:cn=replication dn,orclreplicaid=qdinh-sun_
adeldap,cn=replication configuration-----/

J.1.11.2 Password Not Synchronized

Either oidctl or opmn fails to start an Oracle Internet Directory server instance and the replication server log file oidrepld00.log reports that it is not able to bind.

Problem

The replica DN password stored in the oidpwdrSID is not synchronized with the replica DN password in the Oracle Internet Directory server.

Solution

Try to connect to the Oracle Internet Directory server instance using the ldapbind command. Specify the replica DN stored in oidpwdrSID and the replica DN password. For example:

>> ldapbind -h host -p port -D "cn=replication dn,orclreplicaid=qdinh-sun_adeldap, cn=replication configuration" -w replica_dn_password

If the connection succeeds, then you can reset the password in the oidpwdrSID wallet using remtool with the option -pchgwalpwd, which changes the password of the replication DN of a replica only in the wallet. If you do not remember the replication dn password, then you can reset it using remtool with the option -prestpwd, which resets the password of the replication dn of a replica.

After resetting the replication password wallet, restart the replication server instance again a using opmnctl or oidctl.