Problem:
When a clustered application deployed on a WebLogic 8.1 SP6 cluster tries to connect to a Documentum Content Server during load tests, one or more servers in the WebLogic cluster crash with a core dump. Given below is an extract of the exception stack trace from a JRockit dump file (jrockit.
Stack 0: start=0xf3068000, end=0xf308c000, guards=0xf306d000 (ok), forbidden=0xf306b000 Thread Stack Trace: at
— Java stack —
at com/documentum/dmcl/Dmcl40.get(Ljava/lang/String;)Ljava/lang/String;(Native Method)
at com/documentum/fc/connector/DfConnection.apiGet(DfConnection.java:180)
^– Holding lock: com/documentum/fc/connector/DfConnection@0x1c4a3970[thin lock]
at com/documentum/fc/connector/DfConnection.
at com/documentum/fc/connector/DfConnectionFactory.getConnection(DfConnectionFactory.java:25)
at com/documentum/fc/client/DfClientSupport.getConnection(DfClientSupport.java:623)
at com/documentum/fc/client/DfClientSupport.newSession(DfClientSupport.java:183)
at com/documentum/fc/client/DfSessionManager.newManualSession(DfSessionManager.java:753)
^– Holding lock: com/documentum/fc/client/impl/session/IdentityContext@0x1ce1a2b0[thin lock]
at com/documentum/fc/client/DfSessionManager.createSessionHelper(DfSessionManager.java:627)
at com/documentum/fc/client/DfSessionManager.getSession(DfSessionManager.java:559)
^– Holding lock: com/documentum/fc/client/DfSessionManager@0x1cdfb7c0[thin lock]
at com/documentum/fc/client/DfSessionManager.getSession(DfSessionManager.java:362)
Background & Analysis:
WebLogic Version: 8.1 SP6 (cluster with 2 managed servers)
JVM version: JRockit R27.6 1.4.2_21
DFC version: 5.3 SP6
The JVM crashed whenever the application, under load, attempted to connect to the Documentum Content Server via the Documentum Foundation Classes (DFC) API. When we ran only one server in the cluster (kept the other server in shutdown state), the running server worked fine with no crashes and DFC errors.
Solution:
Upgrade DFC to version 6.0 or higher.
Workaround: If you cannot upgrade to DFC 6 or higher for some reason, then as a workaround, ensure that you have some means of "priming" your application. i.e. initiate connections (to Documentum) from your application (unit testing) on all WebLogic servers within the cluster before your application takes on load. We had to use this workaround as WebLogic 8.1 SP6 is not supported on JVM 1.5+ and DFC 6+ is only supported on JVM 1.5+.
Root Cause:
DFC 5.3 SP6 is not compatible with WebLogic clusters. It is supported only on a single WebLogic node. DFC versions 6 and higher are supported on WebLogic clusters.
NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.
