Ahoy there! This is my personal blog which I use as my memory extension and a medium to share stuff that could be useful to others.

WebLogic: SerializedSystemIniException

Problem:

The WebLogic Server does not start. Following errors seen in stdout/stderr/server logs:

 <21-Jun-2009 08:18:13 o’clock BST> <Warning> <Security> <BEA-090066> <Problem handling boot identity. The following exception was generated: weblogic.security.internal.SerializedSystemIniException: [Security:090207]Version mismatch. have 117, expected 1>

            .

            .

Background & Analysis:

Read this tutorial to understand how a WebLogic server uses SerializedSystemIni.dat and how this file may become corrupted.

 

Solution:

Restore a backup of SerializedSystemIni.dat and start the WebLogic servers.

If you do not have a backup, then you will need to reconfigure the domain.

 

Root Cause:

The WebLogic domain’s SerializedSystemIni.dat file is corrupted.

 

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.9.22_1171]
Rating: +4 (from 4 votes)

WebLogic: JSAFE_PaddingException

Problem:

WebLogic Server 8.1 does not start. Following errors seen in stdout/stderr/server logs:

 ####<Jun 18, 2009 11:44:13 AM BST> <Error> <Management> <> <admin_mkdom> <main> <<WLS Kernel>> <> <BEA-140001> <An error occurred while getting attribute Credential on MBean mkdom:Location=admin_mkdom,Name=mkdom,Type=EmbeddedLDAPConfig. Method: null. Exception: com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte..

com.rsa.jsafe.JSAFE_PaddingException: Could not perform unpadding: invalid pad byte.

            at com.rsa.jsafe.JA_PKCS5Padding.a(Unknown Source)

            at com.rsa.jsafe.JG_BlockCipher.decryptFinal(Unknown Source)

            .

            .

Background & Analysis:

JSAFE is an encryption/decryption engine for Java, developed by RSA. So, you can be sure that you have a problem with encryption/decryption whenever you encounter exceptions related to JSAFE.

All passwords in a WebLogic domain are encrypted by hashes in a file called SerializedSystemIni.dat. Read this tutorial to understand how WebLogic uses SerializedSystemIni.dat. If passwords that have been encrypted in one WebLogic domain are used in another WebLogic domain, then all the encrypted passwords will not be decrypted and the WebLogic servers will not start.

 

Solution:

As the passwords need to be encrypted using the current domain’s SerializedSystemIni.dat file, do the following to configure valid encrypted passwords and start the WebLogic servers:

(1) In all WebLogic configuration files contain encrypted passwords, find and replace as follows:

ALL occurrences of  PasswordEncrypted=”{3DES}…….” 

                                              with

                                Password=”plain-text password”

 

where, plain-text password  =  corresponding plain-text password for encrypted password

Example: If the password for a user used by a connection pool is “olympics” and its encrypted value is “{3DES}hKjgbvd==”, then replace PasswordEncrypted=“{3DES}hKjgbvd==” with Password=”olympics”

(2) Start the WebLogic servers.

 

Root Cause:

The encrypted passwords used in configuration files within a WebLogic domain were encrypted using SerializedSystemIni.dat in another WebLogic domain.

UPDATE (24th January 2010): For WebLogic Server 9.x+, the solution above will not work. Refer this article for tips.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.9.22_1171]
Rating: +9 (from 11 votes)

The SerializedSystemIni.dat file is a critical part of every WebLogic domain and if not administered properly, could result in your servers not starting and you having to recreate the entire domain. So, a good understanding of this file is important.

 

What is it?

SerializedSystemIni.dat is a WebLogic domain file which contains hashes.  Currently, these hashes use Triple-DES block ciphers (that’s why the encrypted passwords begin with “{3DES}”). SerializedSystemIni.dat is located in the domain directory (WebLogic Server 8.1 and earlier) or in domain/security directory (WebLogic Server 9.x and later).

 

When and how is it created?

The SerializedSystemIni.dat is created during the creation of a WebLogic domain. The hashes in the file are created using an algorithm that binds the file to the domain in which it has been created. So, a SerializedSystemIni.dat file can be used only within the domain in which it has been created (cannot be used in other WebLogic domains). Also, in WebLogic Server versions 8.1 SP6 and earlier, the SerializedSystemIni.dat file (along with msi-config.xml and fileRealm.properties files)  is replicated into a managed server’s root directory every 5 minutes for managed servers which have MSI File Replication enabled. These files are replicated even if the admin and managed servers share the same server root directory (doesn’t make sense to me).

 

What is it used for?

SerializedSystemIni.dat is used for encryption/decryption of plain-text/ciphertext within a WebLogic domain.

Gotchas!

  • If SerializedSystemIni.dat is corrupted (modified) or missing, then the WebLogic Servers in your domain will not start and you will have to reconfigure the domain.
  • If MSI File Replication is enabled for your managed servers in domains running on WebLogic server versions 8.1 SP6 or earlier, then a bug in the 5-minute replication could cause corruption of SerializedSystemIni.dat occasionally during server restart and will make it a zero-byte file if the disk drive or mountpoint in which the WebLogic domain is located is 100% full.
  • If SerializedSystemIni.dat is transferred to its domain in ascii mode (for example, from a configuration management system) , the file could become corrupted.

 

Best Practices

  • Ensure that you have a working backup of SerializedSystemIni.dat. i.e. test the recovery of SerializedSystemIni.dat and server start-up using the backup copy.
  • If you’re using WebLogic Server versions 8.1 SP6 or earlier and have MSI File Replication enabled, then ensure you contact Oracle, obtain and deploy patch CR260218 (Guardian Signature Patterns Release 1.1.34 and Signature ID 000168) to fix the bug in replication that corrupts SerializedSystemIni.dat or makes it a zero-byte file.
  • If you require to transfer SerializedSystemIni.dat to its domain via FTP, always use binary mode for the file transfer.
VN:F [1.9.22_1171]
Rating: +42 (from 42 votes)