Problem:

A Java application running on a WebLogic Server uses Messaging Bridges to send messages to JMS destinations on a remote server. The target destinations were configured with the domain name of the remote server. When the remote server changed its IP address and modified DNS entries to map its domain name to the new IP address, the Messaging Bridges could no longer connect to the Remote Server.

 

Background & Analysis:

When a Java application looks up a domain name, the result of the domain name resolution is cached in the java.net.InetAddress class. By default, for Sun Hotspot JVM 1.5.x and earlier, a successful domain name resolution is cached forever (yikes!) and an unsuccessful domain name resolution is cached for 10 seconds. By default, for Sun Hotspot JVM 1.6.x, a successful domain name resolution is cached for 30 seconds (huh, why even bother caching positives only for a few seconds?) and an unsuccessful domain name resolution is cached for 10 seconds.

 

Which parameters control domain name resolution caching?

Either of the following sets of parameters control caching for successful (+ve) and unsuccessful (-ve) domain name resolutions are:

1. networkaddress.cache.ttl and networkaddress.cache.negative.ttl

OR

2. sun.net.inetaddr.ttl and sun.net.inetaddr.negative.ttl

The above parameters represent the time-to-live (expiry period) for java.net.InetAddress cache entries, in seconds. Parameters in (1) are recommended, as parameters in (2) are Sun proprietary parameters and may be removed in future releases.

How to configure the domain name resolution caching parameters?

The networkaddress.cache.ttl and networkaddress.cache.negative.ttl parameters can be configured by setting their values in the ${JAVA_HOME}/jre/lib/security/java.security file.

The sun.net.inetaddr.ttl and sun.net.inetaddr.negative.ttl can be configured by passing them as command-line options to the startup command for the JVM (-Dsun.net.inetaddr.ttl=30)

Sample Code to determine values (especially useful for determining default values):

public class DNScacheTTL
{
  public static void main(String[] args)
   {
     System.out.println(”networkaddress.cache.ttl = 
sun.net.inetaddr.ttl = “+sun.net.InetAddressCachePolicy.get());
     System.out.println(”networkaddress.cache.negative.ttl 
= sun.net.inetaddr.negative.ttl = “+sun.net.InetAddressCachePolicy.getNegative());
   }
}

The above code will throw warnings when compiled with certain JVMs because it uses Sun’s proprietary methods. You may ignore these warnings.

Examples: Code above inserted in DNSCacheTTL.java, compiled and then executed as follows on Sun JVM 1.6:

java DNScacheTTL

networkaddress.cache.ttl = sun.net.inetaddr.ttl = 30

networkaddress.cache.negative.ttl = sun.net.inetaddr.negative.ttl = 10


java -Dsun.net.inetaddr.ttl=100 DNScacheTTL

networkaddress.cache.ttl = sun.net.inetaddr.ttl = 100

networkaddress.cache.negative.ttl = sun.net.inetaddr.negative.ttl = 10


 

Solution:

 

The WebLogic Servers were restarted to clear the java.net.InetAddress caches. Consequently, the messaging bridges looked up the new domain name mapping for the remote server and connected to its target destination successfully.

Root Cause:

 

Domain Name lookup caching in the JVM was configured with default parameters. Hence, old domain name < –- > IP address mapping was cached forever.

 

Tips for setting caching parameters:

 

It’s easy to be tempted to set the networkaddress.cache.ttl value to 0 to turn off caching. However, note that Sun wants you to cache for two reasons – (1) Prevent DNS spoofing and (2) Improve performance. DNS Spoofing will not be an issue if the systems involved communicate over a secure network. So, assuming your systems are part of a secure network, trade-offs must be made between improved performance and service impact (when remote interfaces change – should be a rare occurrence) with caching. If your application is time-critical and cannot even withstand a latency/unavailability of a couple of minutes (e.g. synchronous call with user waiting for response), then turn off caching. However, if your application can tolerate delays by a few minutes or more (e.g. asynchronous message transfers), then you may set the value of networkaddress.cache.ttl to something like 300 (5 minutes). In such a scenario, if a remote system changes its domain name < – > IP address mapping, then your application will connect to the new IP address after 5 minutes, in the worst case. This also ensures that under normal circumstances, if your application makes several connections to the remote server within a 5 minute period, it will lookup the domain only once (assuming successful lookup) in 5 minutes, thereby improving performance. Refer to the table below for some example scenarios. So, hope you get the idea and set networkaddress.cache.ttl suitably. For almost all purposes, the default value of 10 seconds for networkaddress.cache.negative.ttl will suffice.

 

Application Latency Requirement Recommended value for networkaddress.cache.ttl
Low-latency/time-critical 0 (no caching)
Moderate Latency (tolerance of a few minutes) 300 (or depending on latency tolerance)

While considering the above guidelines for setting JVM caching parameters, you must also take into account DNS caching by the Operating System and network devices.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

 

VN:F [1.9.22_1171]
Rating: +14 (from 16 votes)

Tagged with:

Filed under: Middleware

Like this post? Subscribe to my RSS feed and get loads more!