Getting full-end-to-end visibility for critical systems is a “must have” nowadays. Most APM solutions integrate fairly well out of the box with common products such as web servers and application servers, but what about other systems? Unfortunately, not all companies use the latest of the latest cloud based open source technologies or even the big application servers such as IBM, Oracle and Microsoft. Some still use in house custom written applications somewhere in their systems. Where does APM fit?
I recently came across a very interesting case with a customer. Most of their back-end systems were using fairly standard technologies such as Java and Tomcat, but one of the core applications was still running a very simple AWT java-thick client.
This customer is one of the market leaders in online and physical car auctions. The general public can use the standard web site and place bids, but the auction centers still rely heavily on a simple but reliable Java client. Because, this company wanted to be more proactive and not wait for calls from the Clerks when things go bad, having monitoring in all places became a must.
With agent based monitoring, Dynatrace agents can collect all the monitoring information and send it to a central server for analysis.
The first step here was to install the agent on the local machine and make sure that it was “injected” in the application. In our case, the code was delivered via Java web start (AKA JNLP) so it was a little bit more challenging to get it working.
An environment variable called JAVA_TOOL_OPTIONS was used to set the agent path. This way, the customer was able to “soft launch” the new agent by setting the environment variable or not depending on which auction center would go first.
Here is an example of how it was defined:
In the case of non-generic applications, apart from memory and CPU consumption figures not much else can be seen with most tools so it isn’t enough. On a side note, it was quite surprising to see that the JVM was only using 20 MB of RAM maximum where most processes nowadays consume gigabytes!
Dynatrace is compatible with any Java application. It will automatically map transactions to Purepaths (digest representation of what has been executed and where) for most known applications servers. The real power of Dynatrace is that it provides a generic engine to allow users to tailor their configuration to the nth degree. It is therefore possible to “teach” Dynatrace to recognize any system and not just J2EE application servers. This is done, by spending a little bit of time with developers and go through the source code to find the pieces of code which defines a user transaction. Once it had been identified, all we needed was to create a custom sensor rule to indicate where a Purepath should start.
From this point, any bad performing code will be highlighted with auto-sensors.
Auto sensors are a lightweight way of highlighting long running code. They are great because, they don’t require any upfront configuration settings unlike custom sensors. Dynatrace uses both so that’s why it is so powerful. In our case, we only needed two method sensors. One for sending messages and one for receiving messages from the back-end bidding engine. A third method (processEvent) was added for convenience.
Because the code was so simple not much more was required. Each message handler sends the message in text format, so the custom sensor was configured to capture the text because it is a method argument. Now that the data was available, it was easy to build a dashboard which showed activity broken down by auction center.
The customer can now see bids been placed, new auctions starting etc. They no longer require to pick up the phone to check whether or not auctions are running fine.
The screenshot below was taken during the Proof of concept. It was deployed to 2 agents called Hall A (left) and Hall B (right). The upper section shows each message type received by the auction applications and at the bottom, messages sent out such as bid requests. Each type has its own color. The average response time was added in blue to complete the picture.
The only challenge was to define regular expressions to extract the different message types because, unfortunately, the messages were not formatted consistently. This example really shows how fixable Dynatrace can be. Any plug-and-play monitoring solution would not have worked in this case, but with Dynatrace it was possible.