Problem:

When activating changes (clicking the button "Activate Changes" as shown in the image on the left) on the Administration console of a WebLogic 10.0 MP1 domain comprising an admin server and two managed servers (each managed server on a different host), it took around 5 minutes for the activation to complete.

Background:

From WebLogic Server versions 9.x and later, any changes performed on the Administration console must go through a three-step process – (1) Lock and Edit (2) Edit config (3) Activate Changes. It’s the third step in this process that took about 5 minutes to complete. The changes were successfully made, albeit after 5 minutes. Interestingly, when we located all the managed servers in the domain on the same host, this problem disappeared and the activation of changes took less than 10 seconds. However, locating all managed servers on one host cannot be a solution. We enabled debug for Deployment on all servers. Given below is the output of the debug during occurrence of the problem:

####<Sep 29, 2009 10:56:45 AM BST> <Debug> <Deployment> <myhost> <myadmin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1254218205661> <BEA-000000> <Experienced Exception while c.tryLock() and it is ignored :: java.nio.channels.OverlappingFileLockException
at sun.nio.ch.FileChannelImpl.checkList(FileChannelImpl.java:853)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:820)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:967)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:374)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.getFileLock(ConfigDataUpdate.java:357)
at weblogic.deploy.internal.targetserver.datamanagement.ConfigDataUpdate.acquireFileLock(ConfigDataUpdate.java:338)
.
.
.

Solution:

After liasing with Oracle Support, we upgraded our JVM and the upgrade resolved the problem. After the upgrade, the activation of changes took less than 10 seconds irrespective of whether the managed servers were located on the same host or not. Details of the upgrade are given below:

Old JVM:

java version “1.5.0_14″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
BEA JRockit(R) (build R27.5.0-110_o-99226-1.5.0_14-20080528-1505-linux-x86_64, compiled mode)

New JVM:
java version “1.5.0_17″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_17-b04)
BEA JRockit(R) (build R27.6.3-40_BR8141840-120019-1.5.0_17-20090828-1133-linux-x86_64, compiled mode)

Root Cause:

Bug in JVM 1.5.0_14 (JRockit R27.5.0)

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

At times, you may want to determine the elapsed wall-clock time or uptime for a running process. While you can determine the process’ elapsed time using certain versions of the ps utility, it can also be determined using a simple shell script with a bit of perl. Both these methods are described below with the init process (PID = 1):

SOLUTION 1: The ps utility on some variants of UNIX (e.g. Linux, but not Solaris)

ps -oetime 1
    ELAPSED
      39:45

SOLUTION 2: A shell script with a bit of perl (should work on all variants of UNIX)

Download the shell script (puptime.ksh) or copy and paste from below:

#!/bin/ksh
# puptime.ksh - Gavin Satur - http://cybergav.in/2009/11/09/puptime
# Simple script to calculate the uptime (elapsed wall clock time) of a process
##############################################################################
#
# Accept PID and do some basic validation
#
proc_pid=$1
if [ -z "$proc_pid" ]; then
   print "\nERROR : Missing input argument. SYNTAX: ksh puptime.ksh
\n"
   exit 999
fi
if [ ! -d /proc/$proc_pid ]; then
   print "\nERROR : No directory for PID $proc_pid in /proc. Check if process is running!\n"
   exit 999
fi
#
# Calculate start time of process and current time in epoch time using a bit of perl
#
proc_stime=`perl -e 'use File::stat;  my $filename = "$ARGV[0]"; $sb = stat($filename); printf "%s", $sb->mtime;' /proc/$proc_pid`
currtime=`perl -e 'print time;'`
#
# Calculate process uptime in seconds and then slice'n'dice for human-friendly output
#
proc_time=$(( currtime - proc_stime ))
proc_time_days=$(( proc_time / 86400 ))
proc_time_secs=$(( proc_time % 86400 ))
proc_time_hours=$(( proc_time_secs / 3600 ))
proc_time_secs=$(( proc_time_secs % 3600 ))
proc_time_minutes=$((proc_time_secs / 60 ))
proc_time_secs=$(( proc_time_secs % 60 ))
print "\nUPTIME FOR PROCESS WITH PID $proc_pid = $proc_time_days day(s) $proc_time_hours hour(s) $proc_time_minutes minute(s) $proc_time_secs second(s) \n"

./puptime.ksh 1
UPTIME FOR PROCESS WITH PID 1 = 0 day(s) 0 hour(s) 39 minute(s) 37 second(s)

I watched the David Haye Vs Nikolai Valuev WBA Heavyweight title fight a couple of hours ago. There was a lot of hype before the fight billed as “David Vs Goliath”, given the whopping 7 stone weight difference and significant height difference between the two boxers. Before the fight, David promised that he would be the first man to knock out the “beast from the east” and did some thrash talking about the huge Russian.

Well, the fight was far from a spectacle, it was twelve rounds of a cat-and-mouse game, not many punches exchanged and there was no knock out. David Haye won by a majority decision on points and is now the WBA Heavyweight Champion of the world. David was slipping and sliding more than punching. His negative tactics were understandable since he was facing a giant with a massive height and weight advantage, but he could have thrown more punches in the earlier rounds as he in fact rocked Nikolai a bit in the last round with a combination (after the match, David said that he hurt his right hand in the early rounds and so didn’t use it much).

Although no boxing fan would want to see a title being won in such a boring match (and some may even argue that David did not do much in the fight to win the title) , David’s victory is a breath of fresh air for the boxing Heavyweight division. David Haye has the skill, speed, power, aggression and charisma for a world champion (though he can do without a bit of the thrash talking pre-fight), whereas Nikolai is just a boring giant who relies more on his size than skill. Lot of big bucks await David Haye and I wish him well. It will be interesting to watch him take on the Klitshcko brothers.

Problem:

The Administration server of a WebLogic domain comprising WebLogic Server 10.0 and WebLogic Integration 10.2, consumes high CPU and throws java.lang.OutOfMemory errors.

Background:

The WebLogic Domain’s admin server had only two web applications deployed on it – the WebLogic Administration console and WebLogic Integration console. After start-up, its CPU utilization gradually increased and reached around 80% within a couple of days. Also, java.lang.OutOfMemory errors were observed in the server logs. This behaviour was observed even when there was no load on the managed servers and the web applications on the admin server were not accessed (all servers idle from a user perspective).

WebLogic Domain details:

Version: WebLogic Server 10.0 MP1, WebLogic Integration 10.2
JVM: JRockit R27.5.0-110 (JRE Standard Edition build 1.5.0_14-b03)
Admin Server JVM Heap: Minimum (Xms) = Maximum (Xmx) = 2 GB
Number of managed servers: 2
Operating System: 64-bit Red Hat Enterprise Linux 5.1
CPU Architecture: AMD64

Solution:

The following patches were applied and the problem was resolved. Contact Oracle support or use their Smart Update procedure to obtain the patches.

Root Cause:

Known issues with WebLogic Integration 10.2

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

A software’s log4j framework could not find the log4j configuration file which I specified. Details with debug below:

Command:

java -Dlog4j.debug=true -Dlog4j.configuration=/mysoftware/config/mylog4j.properties ...

Debug Output:

log4j: Trying to find [mysoftware/config/mylog4j.properties] using context classloader sun.misc.Launcher$AppClassLoader@164b95.
log4j: Trying to find [mysoftware/config/mylog4j.properties] using sun.misc.Launcher$AppClassLoader@164b95 class loader.
log4j: Trying to find [mysoftware/config/mylog4j.properties] using ClassLoader.getSystemResource().
log4j: Could not find resource: [mysoftware/config/mylog4j.properties].

Background:

I did not want to use the default location for the log4j configuration file and wanted to be able to specify to a file with a name and location of my choice.

Solution:

The debug output indicates that the absolute path of the resource (log4j.properties) is missing a beginning “/”. That’s because the file:// protocol was not specified. So, the following worked and enabled log4j locate the properties file:

java -Dlog4j.debug=true -Dlog4j.configuration=file:///mysoftware/config/mylog4j.properties ...

NOTE: If your log4j configuration file is called log4j.xml or log4j.properties, then placing it in the application classpath will suffice and you will not need the -Dlog4j.configuration option.

Root Cause:

The file:// protocol was not used to specify the log4j configuration file.

Reference:

Short Introduction to log4j – Ceki Gülcü

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.