Ahoy there! This is my personal blog which I use as my memory extension and a medium to share stuff that could be useful to others.

IT Support Archives

Problem:

BlackBerry Exchange Server (BES) 5 users experience intermittent issues with their devices. For example, they receive some emails promptly whereas others after a delay of a few minutes.

The following "Warning" (Event 20709) is seen in the Application Event Logs on the BES 5 Server:

Application Event Log Warning

The following error is seen in the BlackBerry Messaging Agent (MAGT) logs:

[40720] (11/14 00:03:57.796):{0x19E4} MAPI call failed. Error ‘Network problems are preventing connection to the Microsoft Exchange Server computer.’, LowLevelError 6, Component ‘Microsoft Exchange Server Information Store’, Context 1300

Background & Analysis:

  • We were failing over our BES 5 server from primary (site A) to standby (site B).
  • In site A, the BES 5 server connected directly to 1 Exchange CAS node, whereas in site B, the BES 5 server connected to an F5 load balancer which load balanced connections across 2 Exchange CAS nodes (redundancy).
  • Our Exchange Server was on Exchange 2010.
  • BES 5 communicates with Exchange via MAPI/RPC.

 

Solution:

WORKAROUND: Configure BES 5 to communicate directly with an Exchange CAS node, bypassing the F5. Refer BlackBerry KB28704

Possible Solution: I haven’t tested this solution yet, but BlackBerry KB26490 describes a possible solution with a specific F5 configuration (iRules).

 NOTE: Using an F5 load balancer in front of Exchange CAS nodes requires an appropriate configuration of the F5. Refer this detailed F5 deployment guide.

(1) The solution above describes a successful problem-solving experience and may not be applicable to other problems with similar symptoms.

(2) Your rating of this post will be much appreciated as it gives me and others who read this article, an indication of whether this solution has worked for people other than me. Also, feel free to leave comments.

 

VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Given below are three methods (there could be more) of determining your Hostname in PowerShell:

METHOD 1:

# Fastest method
PS C:\> $env:COMPUTERNAME

METHOD 2:

# Slowest method
PS C:\> $(Invoke-Expression hostname)

METHOD 3:

PS C:\> [System.Net.Dns]::GetHostName()

VN:F [1.9.22_1171]
Rating: +6 (from 6 votes)

Disposable email – Use and forget

Have you provided your personal email account when registering on websites using email confirmation? Have you hesitated to provide comments on blogs, forums, etc. because of fear of having to provide your personal email address? Of course, you always have the option of registering a free email account just for this purpose, but this means you have yet another username/password pair to remember or require some time to create a fictional identity. Instead, how about using a temporary email address that requires no registration? That’s exactly what 10 Minute Mail offers!! It provides you with a temporary email address that is valid for 10 minutes (more than enough time to respond to email registrations). Beat the spam caused by various online registrations!

Well, I’ve now found that Disposable Email Addressing (DEA) has been around for some time and there are several DEA services on the web. Refer this blog post for 20 such disposable email services (Mailinator, MyTrashMail, etc.). Website admins are nowadays blocking registrations with temporary email addresses. Well, you can always try!

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Problem:

Solaris hosts indicate high cpu utilization caused by Sun’s CST (cstd.agt) and Net Connect (srsproxy) processes.

 

Background:

Configuration and Service Tracker (CST) and Net Connect are tools provided by Sun Microsystems for proactive system management at a customer site. The software processes launched by these tools run on SOlaris hosts and regularly send data to Sun Microsystems to enable Sun to track system availability and performance and continually improve Sun’s products and services. cstd.agt and srsproxy are processes belonging to CST and Net Connect respectively. While there have been problems/patches for CST and  problems/patches for Net Connect related to high cpu utilization, note that both these tools have reached their EOL and Sun no longer supports them. Instead, Sun has replaced these tools with the Services Tools Bundle (STB).

 

 Solution:

Remove CST and Net Connect software from your Solaris host(s) or replace CST and Net Connect with STB.

 

Root Cause:

CST and Net Connect have reached their EOL and are no longer supported. Hence, using them could lead to high CPU utilization problems.

 

NOTE:

(1) The solution above describes a successful problem-solving experience and may not be applicable to all problems with similar symptoms.

(2) Your rating of this post will be much appreciated. Also, feel free to leave comments.

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

CET – What you see is what your customer sees

Is service fine? What’s the impact to service? These are some of the key questions often asked by IT management in relation to their company’s IT services. In many cases, attempts are made to answer these questions by checking various system health monitors. In rare cases, these questions can be answered accurately by directly asking the customer(s) (although this will give an indication that you’re not in control). So, how do we know a customer’s experience of an IT service? The simple answer – by putting yourself in the customer’s shoes and trying to do what he/she does when using the service. However, implementation of this “customer experience” is not always straightforward, especially in enterprises having complex IT systems. Given below, is an illustration of Customer Experience Testing (CET).

 

CET.png

 

CET is testing a product or service from a customer perspective. CET contributes greatly towards helping you provide an excellent customer experience for your product or service.

Why would Organizations avoid CET?

  • Time
  • Money
  • Complexity

Time and money are the primary reasons why some organizations don’t implement CET even for business-critical services. Some organizations are put off by the complexity of their IT systems. However, this is a big mistake, because if CET is designed and implemented properly, it will actually save time and money.

How do you implement CET?

Design CET: CET is not just a process – it’s also a solution and so must be catered to in solution/system/end-to-end designs. A couple of reasons often given for not testing a particular service like a customer would do, are lack of test data and lack of a process for backend systems/implementers to identify a test. A few years ago, my colleagues and I were working on a problem with the payment fulfilment on a telecom giant’s website. Quite a few customers had reported problems with certain shopping journeys on the website. We wanted to simulate the problem and troubleshoot, but we did not have credit/debit card details to do so and staff were apprehensive of using their own cards due to the cumbersome process involved in getting a refund. We eventually fixed the problem, but had to rely heavily on customer feedback and that would not have made the customer comfortable. If we had test data and an end-to-end CET solution to place an order, then we could have been proactive and identified the fault before the customer reported it and we would have been able to fix the problem confidently and quickly and provide excellent customer satisfaction.

Know your use cases: A well-designed CET solution will use a CET Test Robot – a system that automates interaction with your product or service. In order to ensure that the CET test robot simulates a customer’s interaction with your product/service as closely as possible, you must know all your product’s use cases (what your product/service is designed to cater to) and setup the test robot to execute the use cases and validate responses regularly. Depending on your customer profile, using more than one CET robot and locating them across various locations in which your product/service has customers will provide you a better idea of customer experience.

Know your customer interaction: No matter how much time you spend on identifying use cases, at times, a customer will interact with your product/service in a way that you least expect (and consequently not covered by your use cases). So, how do you simulate these customer interactions in your CET testing? Well, you cannot simulate what you you cannot see. So, you need to know how your customers interact with your product/service. That’s why a solution which records customer interaction across all systems enabling a product/service must be implemented. This solution is sometimes part of “MIS” or “Business Intelligence” initiatives within organizations.

Know your customer: Machines can never completely replace humans (at least I’d like to think so!). While CET Test robots automate use case tests and MIS solutions record customer interaction, there is the possibility of not covering/recording every possible customer interaction. So, that’s why human intelligence is a also a key part of CET Testing – a CET Test Analyst. Rather than just execute a document of standard use cases and recorded customer interactions, the test analyst must think “out-of-box” and try to test the product/service as if he/she were a customer using that product/service. To enable this, the test analyst must work closely with the Support staff (and the customers) to understand the various problems reported by customers and customers’ inclinations. Over time, an experienced CET test analyst will be able to test an almost exhaustive list of customer interactions, thereby playing a vital role in ensuring customer satisfaction.

Use Dashboards: It will be enormously useful to feed the results of CET testing performed by robots and analysts to a dashboard application, so that at any time, the service’s stakeholders can view the status of various services, closely, if not accurately reflecting the usability of these services by the customers. To be useful, the dashboard must be simple, clear and provide the required information about services in an instant to its viewers. You may also choose to make such dashboards public so that in addition to the maintenance pages, you can provide one clear view of the status of all the services your organization provides.

What are the benefits of CET?

  • Tests the end-to-end business process from a customer perspective
  • Gives you a good idea (almost WYSIWYCS – What you see is what your customer sees) of the customer’s experience of a product/service at all times
  • Enables you identify problems with a product/service quickly, take proactive action to notify customers and fix the problem

In large enterprises, CET may not be easy to implement. However, given its huge benefits, CET is a must-have for customer-centric organizations and is definitely a worthwhile investment.

VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Virtualization with Sun VirtualBox

With virtualization so popular, gone are the days when you would need to partition your hard drive and configure dual-boot to give you the option of running more than one OS on your computer.

Sun VirtualBox is a free, open source x86 virtualization software which allows you to create virtual machines (VMs) on your computer’s Operating System (host OS) and run other Operating Systems (guest OS) within the VMs. Sun VirtualBox is easy and intuitive to use and will enable you to quickly test an OS. Well, many OSes nowadays come with LIVE CDs/DVDs to enable you have a test run, but running these OSes in a virtualization software like Sun VirtualBox works better and faster.

Given below is a screenshot of my Sun VirtualBox console. As you can see, I use Sun VirtualBox to run three 64-bit guest OSes (Fedora 11, Solaris 10, Ubuntu 9.04). I have run 2 guest OSes simultaneously, each with 1 GB of allocated memory, without any issues (on my 4 GB Dell Studio XPS 16 laptop).

sun_virtualbox

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Sun VirtualBox is a popular, easy-to-use, free Virtualization software. I use it on 64-bit Windows Vista Ultimate (host OS) to virtualize Solaris10 and a few other Linux distributions. A frequent requirement is to share files and directories between the host OS and guest OSes. Sun VirtualBox makes this sharing easy with the "Shared Folders" feature. Given below are steps I followed to permanently share a directory on my host OS (Windows Vista) with Ubuntu 9.04 (guest OS) on Sun VirtualBox 3.04:

STEP 1: Configure the Shared Folder on the Sun VirtualBox GUI.

Select the Virtual Machine (Ubuntu 9.04) and click "Shared Folders" in the "Details" tab on the right pane. You will see the following window pop-up:

vboxsf

Click the “Add shared folder” icon, select a name for your shared folder, the path to the actual folder on your host OS and the access level for the folder.

I configured the following:

For Ubuntu 9.04 Guest OS:

Name = Ubuntu9.04

Path = E:\VirtualMachines\Ubuntu9.04

Access = Full

For Solaris 10 Guest OS:

Name = Solaris10

Path = E:\VirtualMachines\Solaris10

Access = Full

 

STEP 2: Mount the Shared Folder

For Ubuntu 9.04 Guest OS:

Temporary Mountpoint(disappears after rebooting Guest OS)

Start the Ubuntu 9.04 Virtual Machine, open a terminal window and enter the following command:

sudo mount -t vboxsf Ubuntu9.04 /whatever-you-want

Permanent Mountpoint (remains after rebooting Guest OS)

Start the Ubuntu 9.04 Virtual Machine, open a terminal window and add the following line to the /etc/fstab file:

Ubuntu9.04      /whatever-you-want  vboxsf    rw 0       0

Restart the Ubuntu 9.04 Guest OS.

 

For Solaris 10 Guest OS:

Temporary Mountpoint(disappears after rebooting Guest OS)

Start the Solaris 10 Virtual Machine, open a terminal window and enter the following command as the root user:

pfexec mount -F vboxsf Solaris10 /whatever-you-want

Permanent Mountpoint (remains after rebooting Guest OS)

Start the Solaris 10 Virtual Machine, open a terminal window and add the following line to the /etc/vfstab file:

Solaris10       - /mrkips_vista   vboxfs - yes -

Restart the Solaris 10 Guest OS.

 

References:

UNIX manual (man mount, man fstab)

VN:F [1.9.22_1171]
Rating: +3 (from 7 votes)

Design for Supportability

By Design for Supportability (DFS), I mean designing a product with adequate features  to facilitate administration, support, maintenance  and problem diagnosis. DFS is not just a methodology, it’s a way of thinking for design and development.  This post will pertain to Application Software Design. The basic principle of DFS should be applicable to all product designs.

Benefits of DFS:

  • Increases probability of error detection in the testing phase, thereby leading to a better quality product.
  • Quicker resolution times for incidents and root cause analysis, thereby leading to an increase in customer satisfaction.
  • Smaller support teams, thereby leading to a reduction in the total cost of ownership (TCO).

    Given below are some typical problems experienced by Application Support teams and how they may be addressed/avoided by DFS:

SL# Problem experienced by Support Team How it may be addressed by DFS
1. Inadequate logging. No errors in logs.
  • Use a good logging framework and a well-structured log syntax (eg. log4j) for application logs.
  • Segregate application and server logs.
  • Handle all application exceptions and log descriptive errors, rather than allow errors to be directed to standard error/output.
  • Add debug to critical paths/exit statuses within the application to facilitate troubleshooting problems when debug is enabled.
2.

Logging severity levels cannot be changed at runtime. e.g. to enable debug for the application, the application needs to be redeployed or the underlying server restarted.

  • Use a good logging framework which permits runtime changes (eg. log4j)
3.

Redundant Logging. Rapid utilization of free space on disk.

  • Log adequate details, as required by the severity level. Nothing more, nothing less.
  • Data logged must be in accordance width data privacy/confidentiality laws.
4. Configuration changes are not dynamic.
  • If the application is customizable and driven by configuration files, then as far as possible, dynamic changes to configuration at runtime must be allowed. The more changes to an application that can be made without redeployment/server restart, the better.
5.

Lack of Administration tools. Support teams either perform application administration manually or spend time and money on developing tools.

  • Some applications require ad hoc administrative actions to be performed (eg. process a backlog of messages, resubmit messages, track messages, restart data feeds, etc.) or regular maintenance (eg. housekeeping).
  • Application administration tools must be part of the application deliverable. Spending time and effort on these tools during the design and development stages will pay rich dividends for the in-life support of the application. Also, the designers and developers will be best placed to build such tools.
  • As far as possible, application administration tools must be built to be customizable and automated (eg. as scheduled jobs). The lesser manual intervention by Support teams, the better.

 

How to implement DFS?: If you understand the concept of DFS, you’ll know what’s required to implement it in your project. Here are some guidelines to assist with implementation of DFS.

  • Design for Supportability requires dedicated resources across all teams (design, development, build, testing and support) to work together with utmost cooperation, bearing in mind the ultimate objective – a superb application.
  • Design and Development teams must involve Application Support teams in the early stages of application design (a key requirement for DFS). This will enable the Support team to get an early and good insight into the application and provide feedback with respect to Support requirements. Typically, design and development teams feel it unnecessary to involve Support/Testing teams during the design phase and Support/Testing teams feel it unnecessary to spare resources for the design phase. For DFS to work, this mindset must change.
  • Application Support teams must be involved in the testing phase wherein apart from normal use cases, as many abnormal use case scenarios as possible must be tested. This testing must also cover application administration tools. The Support team must follow a systematic approach to simulate real-world scenarios and test various aspects of the application from a Support perspective and provide feedback to the Development team.

    To all you Application Support folks out there: Spread the word on DFS among your design and development communities. It will make your job easier. DFS should help achieve economies of scale in your Support projects and open the door to new opportunities. Well, there will always be some human presence required (customers won’t want to interact with machines).

VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)