Providing visibility for the Vert-x Framework with Dynatrace

Due to its simple concurrency model and event driven architecture Vert-x is gaining popularity. It allows developers to focus on main application logic and not worry about concurrency control in Java. Like any other Java application, it is not only important to capture every transaction but also have an end-to-end view of transactions running in Vert-x, including all the asynchronous calls. As Dynatrace is the only APM product that captures all transactions and present an end-to-end view, it was time to include Vert-x visibility within Dynatrace. Dynatrace uses byte code instrumentation and auto-sensors to capture transaction details along with stitching the transactions (Purepaths) across multiple tiers automatically. Although Dynatrace covers most Java technologies OOB (Out of the Box), frameworks like Vert-x handle asynchronous calls differently so the default Sensors of Dynatrace are unable to stitch them automatically.

However, that does not mean that Dynatrace customers must wait for future Dynatrace releases to bring visibility to their Vert-x transactions. By combining the power of Dynatrace ADK and AspectJ, you can easily add calls in your code (at runtime) that can connect Purepaths automatically. AspectJ as a tool for AOP has its most prominent use cases where existing Java binaries have to be enriched with additional checks, such as security checks. While the aspects are usually being applied during compile time it is also possible to use load-time weaving, which basically means, that a Java Agent attached to the JVM modifies loaded classes on demand based on the rules defined within the aspects deployed within the class path. This allows the injection of additional calls to the Dynatrace ADK, which in turn helps the Dynatrace Agent determine which transactions are, in fact, simply parts of a larger transaction.

This article does not focus on how to implement aspects in general. The Eclipse AspectJ Project offers plenty of resources and tutorials about this topic.

Creating Purepaths - Custom Entry Points

The Vert-x Framework offers rich functionality for handling HTTP Requests. However, the HTTP traffic is not handled by Servlets in Vert-x, which is supported OOB by Dynatrace. Java classes known by the dynaTrace Sensors. The simplest approach to obtaining a Purepath is to introduce a Custom Entry Point.

For Vert-x 2.5.1 this would be the method:

    org.vertx.java.core.http.impl.ServerConnection.handleRequest(..)

And while this is already a start, it not offer any information about the HTTP Traffic.

A much more favorable solution would be if these Purepaths were actual representations of Web Requests and could also offer the possibility of capturing HTTP Parameters and HTTP Headers. Additionally, these HTTP requests would be categorized by the URL which was requested by the HTTP Client.

The issue here is just that the shipped Dynatrace Sensors are not aware of the objects holding this information, which means there is no OOB support available.

Servlet Invocations

The most feasible way to proceed is to artificially invoke methods that are known to be recognized by Dynatrace OOB. For HTTP and Java the de-facto standard here is the Servlet API.

In this case the invocation of ServerConnection.handleRequest(..) needs to get wrapped in the invocation of method HttpServlet.service(..). More specifically it is required to replace the method body of handleRequest with a call to Servlet that in turn invokes the original method body.

The Dynatrace Servlet Sensor recognizes calls to doFilter(), doGet(), doPost(), doPut(), doDelete(), etc. of class javax.servlet.http.HttpServlet as Web Requests. The details of such a Web Request are available as parameter of type javax.servlet.http.HttpServletRequest. Vert-x offers a similar object of class DefaultServerRequest. Therefore, for proper identification of HTTP calls to Vert-x as Web Request, DefaultServerRequest objects need to get translated into HttpServletRequest objects.

In addition to the invocation of this artificial Servlet also the parameters passed to this Servlet - in particular the HttpServletRequest as defined by the Servlet API - need to get prepared. If they are not delivering proper information about the HTTP request, the dynaTrace Servlet Sensor won’t be able to produce proper Purepaths.

Visualized in pseudo java code this means this method needs to be changed to something like this.

    public class ServerConnection {

      ...

      public void handleRequest(DefaultHttpServerRequest req, DefaultHttpServerResponse res) {
        HANDLE_REQUEST_METHOD_BODY();
      }


      ...

    }

The pseudo call HANDLE\_REQUEST\_METHOD\_BODY() symbolizes the invocation of the method’s body.

AspectJ allows for replacing this method body and offers the possibility to invoke it at a completely different place during the execution flow.

    public class ServerConnection {

      ...


      public void handleRequest(DefaultHttpServerRequest req, DefaultHttpServerResponse res) {
        VertxServlet servlet = new VertxServlet(METHOD_BODY);
        HttpServletRequest hreq = WRAP_REQUEST(req);
        HttpServletResponse hres = WRAP_RESPONSE(res);
        servlet.service(hreq, hres);
      }


      ...

    }

The new method body creates a new Servlet Object, and hands the original method body over to this Servlet.

In order for a Servlet to be properly invoked, there are also artificial instances of HttpServletRequest and HttpServletResponse required. They will provide the Dynatrace Servlet Sensor with the necessary properties of the incoming HTTP request. Their implementation simply translates the request and response objects of Vert-x into Servlet API language.

    public class VertxServlet extends HttpServlet {

      ...

      public void doGet(HttpServletRequest req, HttpServletResponse res) {
        HANDLE_REQUEST_METHOD_BODY();
      }


      public void doPost(HttpServletRequest req, HttpServletResponse res) {
        HANDLE_REQUEST_METHOD_BODY();
      }

      ...
    }

The invocation of method service on a HttpServleteventually leads to doGet, doPost, doPut, … being called.

At this point the Dynatrace Servlet Sensor is able to recognize the HTTP Request and queries for its properties (URL, Parameters, Headers, Cookies, …).

Within these methods the artificial Servlet is now performing the calls the original method body of ServerConnection.handleRequest() contained.

AspectJ allows for replacing the body of a method by defining an Advice around the execution of it. Therefore the invocation of an artificial Servlet is not a real challenge at all here.

Preparing the HttpServletRequest is also relatively easy, because the DefaultHttpServerRequest provided by Vert-x provides the information, although in slightly different data structures. What’s required is an adapter which is able to translate between Servlet API and Vert-x API, in case the Dynatrace Sensors are querying for the request URL or request parameters.

For this approach to work the addition of the classes and interfaces defined by the Servlet API are required. Since Vert-x does not rely on this API, a JAR file containing these additional classes to the CLASSPATH must be added. For the Servlet API the Maven Repository is a good resource.

With this solution in place there is no longer a need for a custom entry point. Simply having the dynaTrace Servlet Sensor placed and configured to active and start PP will suffice.

This approach is feasible for any kind of framework that is capable of handling HTTP traffic, of which Vertx is an example. Furthermore, the solution does not have to impersonate Servlet invocations. If a framework uses HTTP solely for implementing RPC calls, it would be much more beneficial to inject the necessary code for JAX-RS. In this case the Dynatrace Web Service Sensor would handle these calls and create the proper Purepaths and, in theory, would be Asynchronous Request Handling.

However, tracking HTTP traffic is only part of the solution. When working with Vertx the actual request handling happens asynchronously, within a different thread than the one which initially picks up the HTTP request. In a way this is nothing new, since Servlet 3.0 introduced asynchronous processing support. Vertx continues this idea by offering all the required information exclusively via callback handlers since it is rarely the case that a data source can be queried directly for information.

The fundamental part within the Vertx Framework to achieve this is the generic interface org.vertx.java.core.Handler, and is already the case for notifications about incoming HTTP requests, continuously through the whole workflow of handling the incoming data and producing the HTTP response.

Asynchronism is a non-negligible issue when getting full insight into a transaction. What it actually means is that the thread which initially picks up the HTTP request is not necessarily performing all the work required to process it completely. Instead it hands the transaction over to worker threads, freeing the request thread for further requests. The Dynatrace Agent is capable of recognizing these situations provided an application relies solely on Java Thread Pools, for which the Dynatrace Executor Tagging Sensor confirms that a Purepath is produced that covers both the entry thread and worker threads.

This is also, to a degree, how Vert-x manages request handling asynchronously. However Vert-x, to be even more scalable shares worker threads for concurrent requests. At this point the OOB logic of the Dynatrace Sensors will not cover the full transaction - not necessarily for single requests with just small pay.

Stitching Purepaths using the Dynatrace ADK for Java

Fortunately, Dynatrace offers tools to overcome these issues. Using the Dynatrace ADK for Java it is possible to teach the Dynatrace Agent how to bridge the gaps between seemingly unrelated threads.

The Dynatrace ADK is targeted primarily at users who are able to rebuild their own source code, so they can support proprietary wire protocols when sending data from one JVM/CLR to another, or in order to support custom implementations for threading, which works without any issues if you have control over the source code.

For third party libraries this is not always the case. It is still possible to inject the necessary calls to the ADK via AspectJ. The only challenge is to transfer the information about ongoing Purepaths from one thread to another. In the case of Vert-x’s Handler concept the Dynatrace Agent needs to be made aware of the fact that the invocation of a Handler is part of a transaction started within the Thread which registered that Handler. This information needs to be carried along with the Handler Object since it is the only object which is being transferred from one thread to the other.

What the final solution should offer is the OOB ability to visualize which Handlers are being involved when specific URLs are called. In other words, a Method Sensor Rule which covers every invocation of Handler.handle(..), to identify at least a Method Node per invoked Handler object within the Purepath. The screenshot to the left contains that Sensor Rule and excludes several Handler classes which would create too much noise without providing additional insight.

The final Purepaths

The resulting Purepaths now cover most use cases a developer will likely implement when working with Vert-x. It involves asynchronous request handling, asynchronous file handling, the HTTP client offered by the Vert-x Framework and, most importantly, the use of the internal event bus of Vert-x, the mechanism used when sending information from one Verticle to another.

As illustrated in the screenshot on the left, perhaps the most valuable piece of information for every Purepath is the Purepath Duration. It covers the invocations of all involved Handlers while taking into consideration that they are being invoked asynchronously. This property of a Purepath differs significantly from the Purepath Response Time, which focuses only on the execution time of the method which started the Purepath. For a more detailed description of Purepath properties please consult the section about Purepaths within the Dynatrace Documentation.

Conclusion

Similar to the Vert-x example, the Dynatrace ADK can be utilized to gain visibility into any other framework based on Java or .NET where end-to-end ties of transactions are not immediately apparent at first viewing. This applies when stitching together execution paths within the same JVM/CLR, as well as for bridging the gap between multiple JVMs/CLRs that use proprietary remoting / communication protocols to exchange data.

Coverage for publicly-available frameworks like Vert-x will likely be adopted into Dynatrace OOB. Nevertheless, this framework and the solution presented here is a great example of enhancing visibility in cases where OOB coverage by Dynatrace is unlikely due to closed source restrictions. Every third-party library, which is not open source, or even worse, is not available for download publicly, is a potential candidate. This includes in-house solutions developed by customers which, for security reasons, are not shared with Dynatrace or other APM vendors.

In these instances the Dynatrace ADK (also available for .NET and C++) is often the only way to go without having to recompile your libraries.

Source Code Availability

The solution presented here is available on GitHub as part of an effort to implement all the topics referred to within this article.

About Author

Reinhard Pilz

Reinhard has over 15 years of experience as full stack developer in the field of web and APM engineering. In his role as Product Specialist he regularly enables Dynatrace users to get an end to end view and automate their APM process.