WebLogic 10: Activating Changes on the Administration console is slow

Problem:

When activating changes (clicking the button "Activate Changes" as shown in the image on the left) on the Administration console of a WebLogic 10.0 MP1 domain comprising an admin server and two managed servers (each managed server on a different host), it took around 5 minutes for the activation to complete.

Background:

From WebLogic Server versions 9.x and later, any changes performed on the Administration console must go through a three-step process – (1) Lock and Edit (2) Edit config (3) Activate Changes. It’s the third step in this process that took about 5 minutes to complete. The changes were successfully made, albeit after 5 minutes. Interestingly, when we located all the managed servers in the domain on the same host, this problem disappeared and the activation of changes took less than 10 seconds. However, locating all managed servers on one host cannot be a solution. We enabled debug for Deployment on all servers. Given below is the output of the debug during occurrence of the problem:

####<Sep 29, 2009 10:56:45 AM BST> <Debug> <Deployment> <myhost> <myadmin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1254218205661> <BEA-000000> <Experienced Exception while c.tryLock() and it is ignored :: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.FileChannelImpl.checkList(FileChannelImpl.java:853)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:820)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:967)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:374)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:357)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.acquireFileLock(ConfigDataUpdate.java:338)
.
.
.

Solution:

After liasing with Oracle Support, we upgraded our JVM and the upgrade resolved the problem. After the upgrade, the activation of changes took less than 10 seconds irrespective of whether the managed servers were located on the same host or not. Details of the upgrade are given below:

Old JVM:

java version “1.5.0_14”
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
BEA JRockit(R) (build R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-x86_64, compiled mode)

New JVM:
java version “1.5.0_17”
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)
BEA JRockit(R) (build R27.6.3-40_BR8141840-120019-1.5.0_17-20090828-1133-linux-x86_64, compiled mode)

Root Cause:

Bug in JVM 1.5.0_14 (JRockit R27.5.0)

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.