Problem:

HPjmeter 4.0 console does not launch after installation on Windows. When trying to launch it, the Command window briefly appears and then disappears without launching the console.

Background:

HPjmeter 4.0 console, a component of HPjmeter 4.0, is used to view performance data and analyze profiling and GC log files for a JVM. HPjmeter 4.0 console, being developed in java, can be executed on any platform supporting Java. However, the console is currently compatible with only JDK 5 and JDK 6. When you launch the installer (hpjmeter_console_4.0.00_windows_setup.exe), it will first scan your HDDs for installed JDKs and I guess it will pick up the first JDK it finds (see screenshot below).

I have both JDK 1.4.2_11 and JDK 6u17 on my HDD. Although only JDK 6u17 meets HPjmeter 4.0 console’s requirements, the installer selected JDK 1.4.2_11 (must have found this JDK first!) and completed the installation. Well, the installer will not even tell you which JDK it has found and used for the installation. You can determine this information only by checking the hpjmeter.bat file in the bin directory within the installation directory. The JDK used by the installer will be the value of the variable JM_JAVA_HOME in the hpjmeter.bat file. So, for my installation, JM_JAVA_HOME indicated that HPjmeter 4.0 console was using JDK 1.4.2_11, thereby not meeting the software requirements.

The HPjmeter 4.0 console installer should either search and select only a compatible JDK or prompt you for the location of a compatible JDK if it cannot find one. Although the installer isn’t too smart, the product (HPjmeter 4.0) is a very good and robust performance analysis tool for JVMs.

 Solution:

Ensure you have a working JDK 5 or JDK 6 installation. Ensure that the value of JM_JAVA_HOME in the hpjmeter.bat file is the location of the JDK 5 or JDK 6 installation.

Root Cause:

HPjmeter 4.0 console installer detected and used an incompatible JDK during installation. Currently, the HPjmeter 4.0 console is compatible with only JDK 5 and JDK 6.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

Problem:

Solaris hosts indicate high cpu utilization caused by Sun’s CST (cstd.agt) and Net Connect (srsproxy) processes.

Background:

Configuration and Service Tracker (CST) and Net Connect are tools provided by Sun Microsystems for proactive system management at a customer site. The software processes launched by these tools run on SOlaris hosts and regularly send data to Sun Microsystems to enable Sun to track system availability and performance and continually improve Sun’s products and services. cstd.agt and srsproxy are processes belonging to CST and Net Connect respectively. While there have been problems/patches for CST and  problems/patches for Net Connect related to high cpu utilization, note that both these tools have reached their EOL and Sun no longer supports them. Instead, Sun has replaced these tools with the Services Tools Bundle (STB).

 Solution:

Remove CST and Net Connect software from your Solaris host(s) or replace CST and Net Connect with STB.

Root Cause:

CST and Net Connect have reached their EOL and are no longer supported. Hence, using them could lead to high CPU utilization problems.

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

What is MaxTenuringThreshold?:

In a Sun Hotspot JVM, objects that survive Garbage Collection in the Young Generation are copied multiple times between Survivor Spaces before being moved into the Tenured (Old) Generation. The JVM flag that governs how many times the objects are copied between the Survivor Spaces is MaxTenuringThreshold (MTT) and is passed to a JVM as –XX:MaxTenuringThreshold=n , where n is the number of times the objects are copied. The default value of ‘n’ is (or actually was) 31.

Setting the MaxTenuringThreshold:

A few years ago, while my colleagues and I were tuning a 1.4.2_11 Hotspot JVM using flags like PrintTenuringDistribution and tools like  visualgc, we found that setting MTT=10 along with other flags gave us the best results (JVM throughput, pause time, footprint). However, recently when tuning a 1.4.2_11 Hotspot JVM for another application that had mostly short-lived objects, I suggested testing a value of MTT=80 (I still have no idea how the value 80 came to my mind) which is ridiculous as you’ll soon know. My objective was to retain the short-lived objects for as long as possible in the Young Generation to allow them to be collected by Minor GCs as opposed to the Full GCs in the tenured generation. Anyway, all our performance tests of the application on JVM 1.4.2_11 with MTT=80 and other JVM flags showed significant improvement in JVM performance than before (when it was untuned).

Last week, I came across some interesting proposals discussed among Sun engineers last year, regarding modifying the way MTT is handled by the JVM. I don’t know whether the proposals have been implemented, but they give some good insight into how MTT works. To quote those discussions,

Each object has an "age" field in its header which is incremented every time an object is copied within the young generation. When the age field reaches the value of MTT, the object is promoted to the old generation (I’ve left out some detail here…). The parameter -XX:+NeverTenure tells the GC never to tenure objects willingly (they will be promoted only when the target survivor space is full). (out of curiosity: does anyone actually use -XX:+NeverTenure?) Originally, in the HotSpot JVM, we had 5 bits per object for the age field (for a max value of 31, so values of MTT would make sense if they were <= 31). A couple of years ago (since 5u6 IIRC), the age field "lost" one bit and it now only has 4 (for a max value of 15).

Refer http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2008-May/000309.html for more details. So, basically , when I set MTT=80, the JVM would have actually “never tenured” the objects in the Young Generation until the Survivor Spaces were full. Hope Sun have fixed that problem as per the proposal or at least provide proper documentation (similar to the referenced article) for their JVMs which explains how MTT works. Well, MTT=80 did not have an adverse impact on our application, but we eventually switched to MTT=8 (the value 8 was a guess and didn’t provide very different results). I suggest that MTT is not set at first and be used only if your analysis of GC logs and your requirements indicate that you need to retain short-lived objects in the Young Generation for longer. As a matter of fact, when tuning a JVM, always start with basic flags for Heap Size and nothing else. Then, based on load tests, customer experience metrics (e.g. response time, response errors) and analysis of GC logs, set JVM flags and retest. Tuning is iterative and apart from all the tools available, a must-have quality (especially for complex applications) is patience.

Last weekend, I upgraded Windows Vista Ultimate to Windows 7 Ultimate on my Dell Studio xps laptop. I chose to upgrade, rather than perform a complete installation because I have too many programs and a lot of data on my laptop and I didn’t want any overhead in dealing with them. It’s just around seven months since I purchased my laptop and already my DVD drive isn’t working (will need to contact Dell Support and use the warranty). This is 2nd time I’m using the warranty on a Dell laptop to replace a DVD drive (the first time was with my Dell Inspiron 8500), but I still chose to buy Dell, only because Dell provided the most customization options when purchasing the laptop (now running out of patience with Dell).

So, with my DVD drive bust, I needed to perform the upgrade from a USB flash drive. I’ve been using USB flash drives for a few years and since the last couple of years, I’ve stuck with the retractable SanDisk Cruzer Titanium USB flash drives. I like the retractability and the solid build of these drives. I have the 2 GB and 8 GB capacities of the SanDisk flash drive and had to use the 8 GB  flash drive for the upgrade. The upgrade steps I performed are given below:

STEP 1: Choose the correct upgrade edition of Windows 7 Ultimate.

I first obtained the Windows 7 Ultimate N edition and then during installation, a window popped up telling me that I cannot upgrade from Windows Vista Ultimate to Windows 7 Ultimate N and need to do a complete install.

Refer to the Windows 7 upgrade paths to ensure you obtain the correct Windows 7 edition for your upgrade.

STEP 2: Download/Copy the Windows 7 Ultimate ISO image to your hard drive (use SSD if you have one for faster copying).

I downloaded the Windows 7 Ultimate ISO image (en_windows_7_ultimate_x64_dvd_x15-65922.iso – around 3 GB) to my laptop’s HDD.

If you’re downloading the ISO image, ensure that you have sufficient free space in your download location and you turn off your computer’s power saving feature so that your computer does not sleep.

Downloading the ISO frustrated me – my first attempt failed because Google Chrome simply balked after downloading 2.8 GB, my second attempt (using Mozilla Firefox) failed because my laptop went to sleep and I finally downloaded the entire ISO image successfully on my third attempt.

STEP 3: Create a Windows 7 Ultimate bootable USB flash drive

You cannot install/upgrade Windows 7 directly from the ISO. You first need to create Windows 7 bootable media. Popular media are DVDs and USB flash drives (USB 2.0 flash drives are faster than current DVDs for data reading/writing operations and this advantage will significantly increase with USB 3.0). Microsoft provides a free Windows 7 USB/DVD Download tool. If Microsoft removes this tool from their website (I believe there was an issue with using some open source code in the software), then you can get it from other websites or let me know and I can provide you with the tool. Screenshots of the tool are given below:

STEP 4: Prepare for Upgrade

Use the Windows 7 Upgrade Advisor to ensure you meet all the requirements for the upgrade. I had to install Windows Vista Ultimate updates and SP1 to meet the requirements.

STEP 5: Upgrade

Click the “setup” file on your Windows 7 Ultimate bootable flash drive and proceed with the upgrade. You will be required to reboot your computer a few times. I don’t remember how long this process took as I was doing it while half asleep at night. All I know, come dawn, I woke up to Windows 7 Ultimate on my laptop.

My first impressions of Windows 7 Ultimate:

So, I’ve used Windows 7 Ultimate only for a week and haven’t really explored much, but some of the good features that impressed me straight away:

(1) Very fast: Windows 7 Ultimate is the fastest Windows OS I have ever used. Perhaps, running the 64-bit version on a dual-core chip with 4 GB plays a major role, but I can perform routine operations quickly.

(2) Jump Lists: No more looking for “Recent Documents” and thinking about where you stored a specific resource. Just use the “jump list” on the appropriate program and you can jump right away to what you want. Given below is a screenshot of my Adobe Reader jump list displaying the recent pdf files I used.

(3) Snap: Drag windows to screen edges and they resize appropriately. I found this especially useful when using programs that required me to “drag and drop” files from an explorer window into the program’s window (an example of such a program is the Samurai Thread Dump Analyzer).

Well, there is a lot more for me to explore in Windows 7 and I’m specifically interested in Windows 7 Ultimate features like SUA.

Is service fine? What’s the impact to service? These are some of the key questions often asked by IT management in relation to their company’s IT services. In many cases, attempts are made to answer these questions by checking various system health monitors. In rare cases, these questions can be answered accurately by directly asking the customer(s) (although this will give an indication that you’re not in control). So, how do we know a customer’s experience of an IT service? The simple answer – by putting yourself in the customer’s shoes and trying to do what he/she does when using the service. However, implementation of this “customer experience” is not always straightforward, especially in enterprises having complex IT systems. Given below, is an illustration of Customer Experience Testing (CET).

CET is testing a product or service from a customer perspective. CET contributes greatly towards helping you provide an excellent customer experience for your product or service.

Why would Organizations avoid CET?

• Time
• Money
• Complexity

Time and money are the primary reasons why some organizations don’t implement CET even for business-critical services. Some organizations are put off by the complexity of their IT systems. However, this is a big mistake, because if CET is designed and implemented properly, it will actually save time and money.

How do you implement CET?

Design CET: CET is not just a process – it’s also a solution and so must be catered to in solution/system/end-to-end designs. A couple of reasons often given for not testing a particular service like a customer would do, are lack of test data and lack of a process for backend systems/implementers to identify a test. A few years ago, my colleagues and I were working on a problem with the payment fulfilment on a telecom giant’s website. Quite a few customers had reported problems with certain shopping journeys on the website. We wanted to simulate the problem and troubleshoot, but we did not have credit/debit card details to do so and staff were apprehensive of using their own cards due to the cumbersome process involved in getting a refund. We eventually fixed the problem, but had to rely heavily on customer feedback and that would not have made the customer comfortable. If we had test data and an end-to-end CET solution to place an order, then we could have been proactive and identified the fault before the customer reported it and we would have been able to fix the problem confidently and quickly and provide excellent customer satisfaction.

Know your use cases: A well-designed CET solution will use a CET Test Robot – a system that automates interaction with your product or service. In order to ensure that the CET test robot simulates a customer’s interaction with your product/service as closely as possible, you must know all your product’s use cases (what your product/service is designed to cater to) and setup the test robot to execute the use cases and validate responses regularly. Depending on your customer profile, using more than one CET robot and locating them across various locations in which your product/service has customers will provide you a better idea of customer experience.

Know your customer interaction: No matter how much time you spend on identifying use cases, at times, a customer will interact with your product/service in a way that you least expect (and consequently not covered by your use cases). So, how do you simulate these customer interactions in your CET testing? Well, you cannot simulate what you you cannot see. So, you need to know how your customers interact with your product/service. That’s why a solution which records customer interaction across all systems enabling a product/service must be implemented. This solution is sometimes part of “MIS” or “Business Intelligence” initiatives within organizations.

Know your customer: Machines can never completely replace humans (at least I’d like to think so!). While CET Test robots automate use case tests and MIS solutions record customer interaction, there is the possibility of not covering/recording every possible customer interaction. So, that’s why human intelligence is a also a key part of CET Testing – a CET Test Analyst. Rather than just execute a document of standard use cases and recorded customer interactions, the test analyst must think “out-of-box” and try to test the product/service as if he/she were a customer using that product/service. To enable this, the test analyst must work closely with the Support staff (and the customers) to understand the various problems reported by customers and customers’ inclinations. Over time, an experienced CET test analyst will be able to test an almost exhaustive list of customer interactions, thereby playing a vital role in ensuring customer satisfaction.

Use Dashboards: It will be enormously useful to feed the results of CET testing performed by robots and analysts to a dashboard application, so that at any time, the service’s stakeholders can view the status of various services, closely, if not accurately reflecting the usability of these services by the customers. To be useful, the dashboard must be simple, clear and provide the required information about services in an instant to its viewers. You may also choose to make such dashboards public so that in addition to the maintenance pages, you can provide one clear view of the status of all the services your organization provides.

What are the benefits of CET?

• Tests the end-to-end business process from a customer perspective
• Gives you a good idea (almost WYSIWYCS – What you see is what your customer sees) of the customer’s experience of a product/service at all times
• Enables you identify problems with a product/service quickly, take proactive action to notify customers and fix the problem

In large enterprises, CET may not be easy to implement. However, given its huge benefits, CET is a must-have for customer-centric organizations and is definitely a worthwhile investment.