Problem:

A WebLogic managed Server crashed with no relevant information whatsoever in the logs. The server was started by a Node Manager.

Background & Analysis:

WebLogic Version: 8.1 SP6 (cluster with 2 managed servers)
JVM version: 32-bit JRockit R27.6 1.4.2_21
Operating System : 64-bit RHEL 4.0 AS update 7 (kernel 2.6.9)
When the server crashed, the server and stderr logs had no clues regarding the cause of the crash. However, the stdout log was 2147483647 bytes (2 GB) as it was not rotated. The last modified time of the stdout file was the same as the time when the server crashed. The very same scenario was observed when the other server in the cluster crashed. The filesystem is large-file aware.

Solution:

Rotate and archive the stdout file, so that the JVM running WebLogic does not crash when stdout reached 2 GB in size.

NOTE: All logs (server, stderr, stdout, application) must be effectively rotated and archived. I’ve seen several enterprise environments fall victim to lack of log housekeeping. To rotate files like the JVM’s stdout and stderr, it’s best to use the copy-truncate method (make a copy of existing file and then truncate existing file) as the JVM will still have a file descriptor open for the file. You may lose a tiny amount of log information using this method, but it’s less harmful than your server crashing. Removing or renaming a file with an open file descriptor will only make the problem invisible to you as the JVM will still be writing to the old file descriptor and growing a file in a location other than your logs directory (/proc).

Root Cause:

The JVM’s stdout file reached 2GB in size.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

When a clustered application deployed on a WebLogic 8.1 SP6 cluster tries to connect to a Documentum Content Server during load tests, one or more servers in the WebLogic cluster crash with a core dump. Given below is an extract of the exception stack trace from a JRockit dump file (jrockit. .dump):

Stack 0: start=0xf3068000, end=0xf308c000, guards=0xf306d000 (ok), forbidden=0xf306b000 Thread Stack Trace: at (???.c)@0×32553338
– Java stack —
at com/documentum/dmcl/Dmcl40.get(Ljava/lang/String;)Ljava/lang/String;(Native Method)
at com/documentum/fc/connector/DfConnection.apiGet(DfConnection.java:180)
^– Holding lock: com/documentum/fc/connector/DfConnection@0x1c4a3970[thin lock]
at com/documentum/fc/connector/DfConnection.(DfConnection.java:155)
at com/documentum/fc/connector/DfConnectionFactory.getConnection(DfConnectionFactory.java:25)
at com/documentum/fc/client/DfClientSupport.getConnection(DfClientSupport.java:623)
at com/documentum/fc/client/DfClientSupport.newSession(DfClientSupport.java:183)
at com/documentum/fc/client/DfSessionManager.newManualSession(DfSessionManager.java:753)
^– Holding lock: com/documentum/fc/client/impl/session/IdentityContext@0x1ce1a2b0[thin lock]
at com/documentum/fc/client/DfSessionManager.createSessionHelper(DfSessionManager.java:627)
at com/documentum/fc/client/DfSessionManager.getSession(DfSessionManager.java:559)
^– Holding lock: com/documentum/fc/client/DfSessionManager@0x1cdfb7c0[thin lock]
at com/documentum/fc/client/DfSessionManager.getSession(DfSessionManager.java:362)

Background & Analysis:

WebLogic Version: 8.1 SP6 (cluster with 2 managed servers)
JVM version: JRockit R27.6 1.4.2_21
DFC version: 5.3 SP6
The JVM crashed whenever the application, under load, attempted to connect to the Documentum Content Server via the Documentum Foundation Classes (DFC) API. When we ran only one server in the cluster (kept the other server in shutdown state), the running server worked fine with no crashes and DFC errors.

Solution:

Upgrade DFC to version 6.0 or higher.

Workaround: If you cannot upgrade to DFC 6 or higher for some reason, then as a workaround, ensure that you have some means of "priming" your application. i.e. initiate connections (to Documentum) from your application (unit testing) on all WebLogic servers within the cluster before your application takes on load. We had to use this workaround as WebLogic 8.1 SP6 is not supported on JVM 1.5+ and DFC 6+ is only supported on JVM 1.5+.

Root Cause:

DFC 5.3 SP6 is not compatible with WebLogic clusters. It is supported only on a single WebLogic node. DFC versions 6 and higher are supported on WebLogic clusters.

NOTE:
(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.
(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Mutt is a popular email client (MUA) which is common on Linux systems.

Given below are some how-tos on basic uses of mutt. For all UNIX utilities, the "man pages" are your best bet to learn them. I’ve just documented some popular uses of mutt. Refer the "man pages" for a more comprehensive understanding of mutt. The commands below have been tested on Red Hat Enterprise Linux 4.0 AS Update 7 with mutt v1.4.1i, unless otherwise stated.

HOW-TO 1: Send email with blank/empty body

mutt -s "Test email" testmail@abc.com < /dev/null
#
# where:
# -s => Subject
# testmail@abc.com => recipient's email address
#

HOW-TO 2: Send email with body read from a file

mutt -s "Test email" testmail@abc.com < email_body.txt
#
# where:
# -s => Subject
# testmail@abc.com => recipient's email address
# email_body.txt => file containing message body
#

HOW-TO 3: Send email with a customized sender name and email address

# The .muttrc file is Mutt's configuration file. It's default location is the $HOME directory.
# If you locate it elsewhere, specify its location with the '-F' flag.
# Add the following to the .muttrc file:
set realname="Joe Bloggs"
set from="noreply@jb.com"
set use_from=yes
#
# where:
# realname => Sender's name as it appears in the recipient's mail inbox.
# from => the "reply-to" address
#

After configuring .muttrc, send emails as per how-tos 1 and 2.

HOW-TO 4: Send attachment(s)

mutt -s "Test email" -a file1 -a file2 testmail@abc.com < /dev/null
#
# where:
# -s => Subject
# testmail@abc.com => recipient's email address
# file1 => first attachment
# file2 => second attachment
#

HOW-TO 5: Send HTML email

I know that the technical purists out there abhor HTML emails due to potential issues with accessibility and security, but hey, there’s no denying the fact that HTML-formatted emails are far more interesting to look at than plain-text email and are better at drawing your attention to specific information (ask the marketing guys and senior executives!). HTML-formatted emails are supported by Mutt versions 1.5 and higher. Here’s how you may send an HTML-formatted email using mutt v1.5.21:

mutt -e "set content_type=text/html" -s "Test email" testmail@abc.com < welcome.html
#
# where:
# -s => Subject
# testmail@abc.com => recipient's email address
# -e => command to execute
# content_type => email body MIME type
#

The MIME type multipart/alternative ensures your emails are received properly by both plain-text and HTML clients, but it does not work well with mutt at present.

Sendmail is a popular Mail Transfer Agent (MTA) and it is the default MTA on Red hat Linux Enterprise (RHEL). Typically, within enterprises, you will need a Mail User Agent (MUA), an MTA and an SMTP Relay to send emails (outbound) from the Linux command-line or shell scripts.

So, let’s assume that your Mail Administrator provides you IP addresses of primary (192.168.1.1) and secondary (192.168.2.2) SMTP Relay hosts. Given below are steps to configure sendmail (on RHEL 4) to use the SMTP relays to send email (you must use root user’s privileges):

STEP 1: Verify sendmail packages

In order to configure sendmail, you will require the sendmail and sendmail-cf packages. Given below is an example of how to check for these packages:

rpm -qa | grep sendmail
 
sendmail-8.13.1-3.3.el4
sendmail-cf-8.13.1-3.3.el4

STEP 2: Modify sendmail.mc

Do not edit /etc/mail/sendmail.cf. Instead, you must edit /etc/mail/sendmail.mc (more readable than sendmail.cf) and use the m4 macro processor to generate sendmail.cf.

Edit /etc/mail/sendmail.mc and add the following line:

define(`SMART_HOST',`[192.168.1.1]:[192.168.2.2]')

Note:

(1) Pay attention to the quotes. Do not prefix the above by the letters dnl as dnl (delete to new line) denotes a comment in sendmail.mc

(2) If you wish to configure only 1 SMTP relay host (e.g. 192.168.1.1), then add the following:

define(`SMART_HOST',`192.168.1.1')

STEP 3: Start/Restart sendmail

When sendmail is started (if not running) or restarted (if running), then the sendmail.mc file will be processed by the m4 macro processor and a corresponding sendmail.cf will be generated. On RHEL, you may start sendmail as follows:

/sbin/service sendmail start

STEP 4: Use an MUA to test outbound mail

In order to test your sendmail configuration and the SMTP relays, use an MUA to send emails. Given below is an example that uses mutt to send emails:

mutt -s "test email" test@abc.com < /dev/null

Unlike the Oracle client which provides you with the required drivers and tools (e.g. sqlplus) to execute SQL statements against an Oracle database on a variety of platforms, Microsoft does not have such drivers and clients for non-Microsoft platforms. For example, there is no Microsoft driver which you can install on RHEL 4 to allow you configure an SQL interface to Microsoft’s SQL Server.

The goal of ODBC is to solve this very problem by providing a standard software interface for accessing a variety of database management systems on a variety of platforms.

Given below are steps to configure an SQL interface with MS SQL Server on a Red Hat Enterprise Linux 4 (RHEL 4) host:

STEP 1: Verify the installation of unixODBC

unixODBC is provided with Linux by default for recent versions of most Linux distributions. Refer the unixODBC home for more details on unixODBC

rpm -qa | grep unixODBC

If the unixODBC packages are not installed, then download them and install them using rpm

STEP 2: Install FreeTDS

Refer the FreeTDS home for more details on FreeTDS. Download the latest stable release and install it using rpm

STEP 3: Configure a DSN

In order for the unixODBC software to interface with MS SQL Server, the relevant database access details must be provided in a Database Source Name (DSN). To configure a DSN, edit /etc/odbc.ini and add the relevant details. Given below is a sample DSN in /etc/odbc.ini :

[MY_DSN]
Driver=/usr/lib64/libtdsodbc.so.0
Server=192.168.1.1
Port=1433
Database=TEST_DB
TDS_Version=7.0

In the sample DSN above, /usr/lib64/libtdsodbc.so.0 is the absolute path to the FreeTDS driver.

STEP 4: Use isql to interface with MS SQL Server

After successfully completing the 3 steps above, you are now ready to perform operations on the database. To do so, you may use the isql utility that is bundled with the unixODBC package. Given below are few examples on using isql :

# The examples below assume you have a DSN called MY_DSN
# configured in /etc/odbc.ini and the isql utility in your PATH.
#
# Example 1 : Open an interactive session
#
isql MY_DSN username password
 
+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| quit                                  |
|                                       |
+---------------------------------------+
SQL>
#
# Example 2 : Execute an SQL statement. Assume the statement is in a file
# called test.sql. The last line in test.sql must be a blank line.
#
cat test.sql | isql MY_DSN username password

Refer the isql man page for more details on using the isql utility.