A Troubleshooting OC4J

A.1.1 Unable to Start OC4J When Using JDK 1.3

Problem

OC4J fails to start when using JDK 1.3.

Solution

The failure to start is caused by a logging implementation dependency issue. The solution to this problem is to remove or comment out the following entry in ORACLE_HOME/j2ee/home/config/server.xml:

<j2ee-logging-config path="./j2ee-logging.xml" />

A.1.2 Unable to Restart OC4J After Abnormal Termination When OracleAS JMS is Active

Problem

When persistence is enabled in OracleAS JMS, the JMS server creates persistent queues/topics. It also creates lock files (.lock) associated with these queues/topics in the /persistence directory. If the JVM is terminated abnormally, such as with kill -9, the lock files are not deleted. This creates a condition in which OC4J cannot be restarted.

Solution

Manually delete all.lock files from the /persistence directory.

A.1.3 Stateful Replication Not Consistent Across OC4J instances

Problem

The common scenario is that failover is seen from OC4J instance A to instance B, but not back again from B to A.

Solution

OC4J does not require stateful replication to be set up globally for all applications, but instead allows each Web module to configure replication through the <cluster-config> element in its orion-web.xml descriptor file. Ensure that this element is populated correctly in each Web module's descriptor.

Note that OPMN-managed OC4J only supports clustering at the global level, and not at the application/module level.

A.1.4 Using A Non-Certified Version of the JDK for OC4J Only

Problem

In this scenario, you wish to use a later version of the JDK with OC4J than the version certified for use with all Oracle Application Server components. However, using a later version of the JDK globally for all components increases the risk of breaking certification.

Solution

To use the later JDK version with OC4J only, specify its location in the <java-bin> element in the opmn.xml configuration file. For example:

<module-data>  <category id="start-parameters">
    <data id="java-bin" value="/myjavalocation/jdk/bin/java"/>
  </category>
</module-data>

A.1.5 java.lang.OutOfMemory Errors Thrown When Running OC4J

Problem

This error indicates that the heap size of the Java instance is lower than the memory required by applications running within OC4J.

Solution

Increase the heap size by setting -Xmx to the desired amount of memory in the <java-option> element in opmn.xml:

<module-data>  <category id="start-parameters">
    <data id="java-options" value="-Xmx256M" />
  </category>
</module-data>

Alternatively, you can set a system property at OC4J startup:

java -Xmx256M -jar oc4j.jar

If running under Unix/Linux, verify that ulimit settings allow the JVM process to allocate this much memory.

A.1.6 Connection Timeouts Through a Stateful Firewall Affect System Performance

Problem

To improve performance the mod_oc4j component in each Oracle HTTP Server process maintains open TCP connections to the AJP port within each OC4J instance it sends requests to.

In situations where a firewall exists between OHS and OC4J, packages sent via AJP are rejected if the connections can be idle for periods in excess of the inactivity timeout of stateful firewalls.

However, the AJP socket is not closed; as long as the socket remains open, the worker thread is tied to it and is never returned to the thread pool. OC4J will continue to create more threads, and will eventually exhaust system resources.

Solution

The OHS TCP connection must be kept "alive" to avoid firewall timeout issues. This can be accomplished using a combination of OC4J configuration parameters and Apache runtime properties.

Set the following parameters in the httpd.conf or mod_oc4j.conf configuration files. Note that the value of Oc4jConnTimeout sets the length of inactivity, in seconds, before the session is considered inactive.

Oc4jUserKeepalive on

Oc4jConnTimeout 12000 (or a similar value)

Also set the following AJP property at OC4J startup to enable OC4J to close AJP sockets in the event that a connection between OHS and OC4J is dropped due to a firewall timeout:

ajp.keepalive=true 

For example:

java -Dajp.keepalive=true -jar oc4j.jar

A.1.7 OPMN-Managed OC4J Unable to Access EJB Resources Via the Default RMI Port

Problem

OC4J cannot access EJB resources via the default RMI port when running as a component of Oracle Application Server.

Solution

The most common cause is that a user more familiar with Standalone OC4J is reading the RMI port from rmi.xml, unaware that the value specified in this file is not used in an OPMN-managed Oracle Application Server environment.

OPMN-managed OC4J instances use dynamic RMI port assignments. The port value ranges are specified in the <port> element in opmn.xml or are determined using dynamic opmn:ormi lookup from the application client.

See the Oracle Process Manager and Notification Server Administrator's Guide for more information.

A.1.8 Application Performance Impacted by JVM Garbage Collection Pauses

Problem

An application running on OC4J appears unresponsive, with simple requests experiencing noticeable delays. The cause is that the JVM has crossed the low memory threshold and is running a full garbage collection to free up memory.

Solution

Consider using the incremental low pause collector, which avoids long major garbage collection pauses by doing portions of the major collection work at each minor collection. This collector (also known as the train collector) collects portions of the tenured generation - a memory pool holding objects that are typically collected in a major collection - at each minor collection. The result is shorter pauses spread over many minor collections.

Note that the incremental collector is even slower than the default tenured generation collector when considering overall throughput.

To use the incremental collector, the -Xincgc option must be passed in on the Java command line at application startup. Set the initial and maximum size of the young generation (object pool) to the same value using the XX:NewSize and -XX:MaxNewSize options. Set the initial and the maximum Java heap sizes to the same value using the -Xms and -Xmx options.

For example, to use this collector with a server with 1GB of physical memory:

java -server -Xincgc -XX:NewSize=64m -XX:MaxNewSize=64m -Xms512m -Xmx512m

For more information on garbage collection tuning, read "Tuning Garbage Collection with the 1.4.2 JavaTM Virtual Machine" which is available at http://java.sun.com/docs/hotspot/gc1.4.2/

A.1.9 Invalid or Unneeded Library Elements Degrade Performance

Problem

If the OC4J process memory is growing consistently during program execution, then you may have references to invalid symbolic links in your global application.xml file.

This problem is usually characterized by a growth in the C heap and not a growth in Java object memory, as one would see with a more traditional Java object memory leak. OC4J loads all resources defined in the application.xml file. If these links are invalid, then the C heap continues to grow, causing OC4J to run out of memory.

Solution

Ensure that all symbolic links are valid in application.xml, and restart OC4J.

In addition, keep the number of JAR files OC4J is configured to load to a minimum. Eliminate all unused JAR files from the configuration and from the directories OC4J is configured to search. OC4J searches all JAR files for classes and resources, thereby causing the file cache to use extra memory and processor time.

You can control the loading more precisely if your <library> elements in the application.xml file point to the individual JAR and ZIP files that are needed, instead of to the directories where they reside.

A.1.10 JSP Error: Tag Not Registered

Problem

This error occurs when a JSP attempts to call a tag that cannot be found within the OC4J server. The problem typically arises when one or more additional "well-known tag library locations" have been incompletely defined within OC4J.

Solution

Defining a well-known tag library location is a two-step process:

• The directory is defined in the jsp-taglib-locations attribute of the of the <orion-web-app> element in the global-web-application.xml file; and

• The directory is added to the path attribute of the <library> element in the application.xml.

The error typically indicates that the second step was not completed.

Alternatively, you can copy the JAR file containing the tag library to the default well-known tag library location, which is ORACLE_HOME/j2ee/home/jsp/lib/taglib/.

A.1.11 JSP Error: Zero-Length Class File

Problem

This error occurs when OC4J attempts to serve a JSP that failed to compile into a Java class. The cause is that the Java compiler either could not be loaded or ran out of memory, resulting in a 0 byte .class file.

Solution

Delete the 0 byte .class file. The class will be compiled the next time the JSP is requested.

A.1.12 JSP Error: Illegal use of <when>-style tag without <choose> as its direct parent

Problem

This error occurs when OC4J fails to serve a requested JSP that includes a JSP Standard Tag Library (JSTL) tag—in this case, the <choose> tag. The likely cause is that more than one version of the JSTL exists within the OC4J instance.

Solution

Delete the version of the JSTL installed by default with OC4J. This library is packaged as the standard.jar file in the ORACLE_HOME/j2ee/home/jsp/lib/taglib directory.