Update: Dynatrace now includes fully automated support for Go-based application monitoring with our OneAgent. Learn more about Go application performance monitoring.

Providing fast feedback is a key aspect of continuous integration, so kicking off a new build when changes are committed is a common practice in modern software development. One cool visualization of the current builds (trunk and latest sprint) are our Pipeline State UFOs, located around our development office to provide our engineers with fast — and highly visible — feedback no matter where they are in the building, even when they are grabbing a coffee and realize that they may have just broken the build!

Build feedback has to be fast and visibile: Dynatrace Pipeline State UFO
Build feedback has to be fast and visibile: Dynatrace Pipeline State UFO

In recent weeks I had the interesting task of improving Dynatrace’s CI build-triggering process which ultimately feeds the Pipeline State UFO (get it on GitHub). Although it fulfilled basic requirements, like stability and appropriate speed, there were some problems with regard to configuration, versioning or maintainability. For example, a part of the logic runs as Groovy code on the CI server, and connecting the different tasks relied on shell scripts with curl commands.

The redesign of the process using new technologies was the perfect task for my internship at the Dynatrace Test Automation (http://jobs.dynatrace.at/) team. To solve all the mentioned problems a new architecture was necessary. So the primary target was to get the Groovy code out of the CI server, and split logic and configuration. There was also an opportunity to broaden my knowledge of modern programming languages, because I able to use GO (https://golang.org/) for the logic component.

What is GO & Why use it?

For those who haven’t heard of GO — also called Golang — it is a native, open-source language created at Google, partly developed by the famous Ken Thompson (https://en.wikipedia.org/wiki/Ken_Thompson), co-creator of Unix. Its approach is to get the best out of other languages, but also to resolve their shortcomings. As a result, GO is a compiled, statically-typed language with concurrent programming features, garbage collection and more. It can target various platforms, including smartphones. Go is in use at Google, Netflix, Dropbox, Uber and SoundCloud, and Docker is written completely in GO.

In this example I wrote a GO application that was dealing with the logical build steps while it was reading its configuration from a JSON-formatted file. I also wanted to add automated monitoring of my GO executions when triggered by our VCS server.

Why Monitoring GO Executions?

You may be wondering why it is important to monitor this application? Although my GO code might be not very complex, it gets executed every time somebody checks in code. And there are things that could go wrong, such as parsing the incoming file from the VCS, which could possibly be malformed due to some bug earlier in the process. Or, someone made a configuration mistake so that wrong builds would be triggered. Perhaps someone wanted to trigger builds while the CI server is in maintenance and not reachable. As you see, there are several things that could potentially go wrong.

Logging everything that is (or might be) wrong appears as the standard solution, but would produce a high volume of log output when building every minute! The log output would be quite huge, maybe confusing, and would require some type of log analytics to correlate errors with build numbers.

Because I use Dynatrace AppMon I can obtain the troubleshooting information much easier without going through logs, Splunk or ElasticSearch. I simply leverage the PurePath Technology for each incoming build request and capture parameters as well as return values and errors. To find errors in the process, we can filter the PurePaths by the VCS revision number of the commit with the corrupted output using a Business Transaction. Here is a sample output showing me every Revision (Splitting Column) with its Execution Time (in ms) and whether the execution has failed or not.

Splitting shows us the revision number and with Dynatrace’s filter functions, it is easy to locate the one we need.
Splitting shows us the revision number and with Dynatrace’s filter functions, it is easy to locate the one we need.

We can also generate a chart, which shows us the number of incoming build requests over a specific time interval.

Number of builds can also be seen by charting the number of Go app executions – great to validate what our build servers tell us.
Number of builds can also be seen by charting the number of GO app executions – great to validate what our build servers tell us.

Instrumenting your GO App with Dynatrace AppMon

If you want to try this please download the Dynatrace AppMon & UEM Free Trial and start monitoring your own GO application! After the initial 30-Day Trial Period you can keep using Dynatrace to analyze your local apps – FOR LIFE!

Next, I will show you the necessary steps to analyze any GO Application. If you have other types of Apps such as Java, .NET, PHP, Node.js, Mobile or Web simply follow the guides as shown in the Video Tutorials we host on YouTube.

Dynatrace Native ADK

Dynatrace has agents for multiple platforms but, unfortunately, not for native GO applications. For these cases, you can use the so-called Native ADK. The Native ADK provides C and C++ functions for instrumenting the source code. You can either work with the provided macros or directly with the functions of the ADK. I prefer the second approach, because you have more control about how to monitor your application, even if you must sometimes obtain some values manually.

Using the Native ADK

You may be wondering about how to get C code into GO code. Fortunately, GO introduces the Cgo command, which allows GO packages to call C code that is written directly above an import of the pseudo-package “C”. To separate code it is also possible to put the C code in its an own C file, write a header file and import it.

The only requirement to use the feature is an installed GNU C Compiler. For Windows systems you can for example use MinGW. Of course there are some differences between C and GO, particularly when it comes to data types, but Cgo provides you with the necessary converter functions between C and GO types.

With that knowledge it’s easier to start working with the Native ADK. You only need the ADK and agent libraries on your system (install Dynatrace or put them into your GO workspace if you have them), set the CFLAGS and LDFLAGS with pseudo #cgo directives, and also set the needed environment variables. That’s all you need to start creating your PurePath!

So all together your result should for example look like this:

CGO

If you put the ADK and agent libraries into your src folder, you can simple write for example ${SRCDIR}/adk/include, which would be expanded to /go/src/projectname/adk/include.
If you put the ADK and agent libraries into your src folder, you can simple write for example ${SRCDIR}/adk/include, which would be expanded to /go/src/projectname/adk/include.

Environment variables

For Windows systems you have to use the variable PATH instead of LD_LIBRARY_PATH.
For Windows systems you have to use the variable PATH instead of LD_LIBRARY_PATH.

Monitoring GO Applications

The Native ADK provides the basic functions to create a PurePath, together with capturing parameters, return values, exceptions, log entries and more.

In the provided files “nativeAdkFunctions.h” and “nativeAdkFunctions.c” (download from GitHub) there are most of the functions you will need. You can edit the two constants FILE and API in the header file to your preferred labels to get started.

You can find all the necessary information about these functions in the Native ADK documentation but, as mentioned above, I used the Native ADK functions directly instead of the macros. Here is an explanation on how to use the functions in the given file.

Agent

The initializeAgent() function initializes the Dynatrace Agent, while the uninitializeAgent() function terminates it and shuts it down. So both functions should be called only once per application – on startup and shutdown.

Insert6

Sensor placement

It’s quite the same with placeSensor() and exitSensor(), but this time on a per-function basis. Just pass the method name and the type of sensor placement (0 – simple, 1 – entry point) to placeSensor() and you’ll receive a struct with a method id and a serial number. You’ll need them for the other functions that are executed in the context of the function, as it is the logError() function in this example. To mark the end point of the sensor, just call exitSensor() with the previously received struct as parameter. Alternatively, you can also call one of the return*AndExitSensor() functions, which basically does the same thing, but also captures a return value from the given type to show it in the PurePath.

Insert7

Tagging

Tagging is necessary to interact with agents on other systems and build a connected PurePath. Each new connection also needs a new tag, so it has to be fetched before each corresponding call. For example, with a REST call it would probably look like this:

Note that in addition to call C.linkClientPurepath() you also have to put the tag with the key “X-Dynatrace” into the request header.
Note that in addition to call C.linkClientPurepath() you also have to put the tag with the key “X-Dynatrace” into the request header.

Troubleshooting

If you experience timeouts or corrupted paths, please check if you end every placeSensor() with a corresponding exitSensor() with the correct context. The same of course is true for initializeAgent() and uninitializeAgent(). Tagging requires a running agent on the targeting system. GO likes to switch threads by itself, which leads to sometimes losing the tag. In these cases you have to lock and unlock the thread before a critical section with the functions provided by the “runtime” package (for example runtime.lockOSThread()).

Resulting PurePath

If you configured your application correctly, you will see a PurePath starting in your GO app and potentially continues in whatever external system you call. In my case – as shown in the screenshot – I have a PurePath starting at my incomingRequest method that captures context information as method arguments and return values. I also get to see the PurePath continue into the build server which is also instrumented with a Dynatrace Agent.

PurePath starting in Go and connecting into external systems such as the CI Server in my case.
PurePath starting in Go and connecting into external systems such as the CI Server in my case.

That would be what you see if everything goes well, but what if something goes wrong? Here is an example of the PurePath from when our build server was down for maintenance. The return value of the sendBuildRequest() function is the response status code – 404 – which is a nice indicator that the request hasn’t reached the server. In the case of errors, there will be log messages in the PurePath with a short description of what happened or where it happened.

Context information in the PurePath makes it easy to identify what is going on while my code gets executed.
Context information in the PurePath makes it easy to identify what is going on while my code gets executed.

Conclusion

The increasing use of GO, especially at Google, didn’t come out of nowhere. It is a nice, well-structured language providing everything needed to write applications, regardless of complexity, and may replace many C programs in the future. Multiple IDEs and text editors already support GO, which makes developing more comfortable.

Monitoring a GO application may seem a bit complex at first, but with the Native ADK and cgo there really isn’t any complexity at all. Just include the provided C code, use the installed Dynatrace ADK and agent libraries, place sensors in your methods, capture parameters or return values if you want to and you’ll find your PurePath in Dynatrace! With Business Transactions, you have every possibility to analyze the application in any way you like.

More Resources