Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

***************************************************************************
The WebLogic Server did not start up properly.
Exception raised: ‘weblogic.server.ServerLifecycleException: Server failed to bind to the requested port. See preceeding log message for details.’
Reason: Server failed to bind to the requested port. See preceeding log message for details.
***************************************************************************

Background & Analysis:

In order for a software process to listen at a port on a specific IP address, there should not be any other process already listening to that same port and IP address.

Solution:

 (1) Obtain details of the IP address (ListenAddress) and port(ListenPort) at which your WebLogic Server is supposed to listen.

(2) If the host bearing the IP address is unreachable, then contact your Network/System Administrators to ensure that the IP address is associated with a network interface card on the appropriate host.

(3) If the IP address is reachable, use the following commands on the host bearing the IP address (obtained in above step) to determine whether a process is already bound (listening) at the same port and IP address:

On Windows Systems (DOS prompt):

netstat –an | findstr “<IP address>:<port>.*LISTEN”

On UNIX Systems:

netstat –an | grep “<IP address>[\.:]<port>.*LISTEN”

If no software process is listening at the specified IP address and port, then the above commands should not display any output.

Examples:

(a) IP:Port not bound: For a Windows host on which no process listens at port 777 on 110.120.130.140,

C:\netstat -an | findstr "110.120.130.140:777.*LISTEN"

C:\

(b) IP:Port already bound: For a Solaris host on which a WebLogic Server is listening at port 777 on 110.120.130.140,

$ netstat -an | grep "110.120.130.140[\.:]777.*LISTEN"
110.120.130.140.777        *.*               0      0 131072      0 LISTEN

(4) If step (3) indicates that a process is already listening at the port and IP address at which you want your WebLogic Server to listen, then that process needs to be killed (if it’s not meant to be listening there).

(5) After ensuring that no other process is listening at the WebLogic Server ListenAddress and ListenPort, start the WebLogic Server.

NOTE:

• To obtain the IP address for a domain name, use nslookup (other utilities also available).
• If your WebLogic Servers are running within a Veritas Cluster Server (VCS) High Availability framework, then the IP address for your servers will be floating and you must execute the above commands on all hosts within the Veritas cluster.
• This problem and suggested commands are not specific to WebLogic Server processes, but are applicable to all software processes.

Root Cause:

For the problem I experienced, the IP address of the host on which a WebLogic Server was supposed to listen, went missing (error in VCS config caused the IP address to drop off the network interfaces on all the hosts within the VCS cluster) and consequently, was not reachable.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

 <21-Jun-2009 08:18:13 o’clock BST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090207]Version mismatch. have 117, expected 1>

            .

            .

Background & Analysis:

Read this tutorial to understand how a WebLogic server uses SerializedSystemIni.dat and how this file may become corrupted.

Solution:

Restore a backup of SerializedSystemIni.dat and start the WebLogic servers.

If you do not have a backup, then you will need to reconfigure the domain.

Root Cause:

The WebLogic domain’s SerializedSystemIni.dat file is corrupted.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

WebLogic Server 8.1 does not start. Following errors seen in stdout/stderr/server logs:

 ####<Jun 18, 2009 11:44:13 AM BST> <Error> <Management> <> <admin_mkdom> <main> <<WLS Kernel>> <> <BEA-140001> <An error occurred while getting attribute Credential on MBean mkdom:Location=admin_mkdom,Name=mkdom,Type=EmbeddedLDAPConfig. Method: null. Exception: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte..

com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte.

            at com.rsa.jsafe.JA_PKCS5Padding.a(Unknown Source)

            at com.rsa.jsafe.JG_BlockCipher.decryptFinal(Unknown Source)

            .

            .

Background & Analysis:

JSAFE is an encryption/decryption engine for Java, developed by RSA. So, you can be sure that you have a problem with encryption/decryption whenever you encounter exceptions related to JSAFE.

All passwords in a WebLogic domain are encrypted by hashes in a file called SerializedSystemIni.dat. Read this tutorial to understand how WebLogic uses SerializedSystemIni.dat. If passwords that have been encrypted in one WebLogic domain are used in another WebLogic domain, then all the encrypted passwords will not be decrypted and the WebLogic servers will not start.

Solution:

As the passwords need to be encrypted using the current domain’s SerializedSystemIni.dat file, do the following to configure valid encrypted passwords and start the WebLogic servers:

(1) In all WebLogic configuration files contain encrypted passwords, find and replace as follows:

ALL occurrences of  PasswordEncrypted=”{3DES}…….” 

                                              with

                                Password=”plain-text password”

where, plain-text password  =  corresponding plain-text password for encrypted password

Example: If the password for a user used by a connection pool is “olympics” and its encrypted value is “{3DES}hKjgbvd==”, then replace PasswordEncrypted=“{3DES}hKjgbvd==” with Password=”olympics”

(2) Start the WebLogic servers.

Root Cause:

The encrypted passwords used in configuration files within a WebLogic domain were encrypted using SerializedSystemIni.dat in another WebLogic domain.

UPDATE (24th January 2010): For WebLogic Server 9.x+, the solution above will not work. Refer this article for tips.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

WLtd is a simple korn shell script that will enable you manage thread dump operations on WebLogic Servers efficiently and effectively. The salient features of WLtd are given below:

• Can be run in interactive and non-interactive modes (so can be run as a cron job)
• Configuration-driven
• For more than one specified WebLogic Server, thread dumps are taken parallelly (as coprocesses – beneficial on multi-cpu hosts).
• Thread dumps are extracted from the WebLogic Server stderr logfiles and stored in text files.
• Option to email thread dumps, as an attachment (text files will be archived and compressed).
• Housekeeping of thread dump text files

System Requirements:  Solaris/Linux, Korn Shell (/bin/ksh)

Download WLtd v1.0                 Download WLtd v1.0 ReadMe