Dynatrace is a true player in the DevOps tool chain. We built Dynatrace to provide fast feedback on code quality all the way from Dev, via Continuous Integration/Continuous Delivery, all the way into production. The same Dynatrace solution for every pipeline stage to provide fast metrics-based feedback loops on code changes that you are pushing from left (Dev) to right (Ops).

Shift-Left therefore really means that we want to identify bad code quality early on. We also re-define code quality as not just meeting functional specification. We look into performance, scalability and architecture quality. Many of our users execute Unit-, Web/REST API-, Functional- or Load Tests automatically through Bamboo, Jenkins, TFS or other build servers. Dynatrace automatically levels-up these tests by identifying performance, architectural and scalability regressions across builds introduced by the recent code changes. We do this by automatically analyzing and baselining key application metrics captured while you execute your tests against your application as part of your automated build and pipeline execution.

If a regression is detected, e.g: a code or configuration change results in an increase of SQL Queries, and increase in Microservice calls or changed resource consumption behavior then Dynatrace tells the build server (Jenkins, Bamboo, TFS, …) to stop the build early (left) in the lifecycle. Giving the developers faster feedback on critical quality metrics:

Dynatrace enables real Shift-Left by detecting performance-, scalability and architectural regressions when code changes are pushed through your pipeline.

Shift-Left with Bamboo and Dynatrace

While we often talk about users that integrate Dynatrace into Jenkins we see more and more that choose other solutions. A very prominent one is Bamboo, the CI/CD solution from Atlassian.

The Dynatrace AppMon Plugin for Bamboo has been available for a while now – thanks to Jan Swaelens from Sofico who built the integration based on the open Dynatrace AppMon Test Automation REST API. You can watch this Shift-Left YouTube Tutorial to see how Jan is using Bamboo with Selenium and Dynatrace AppMon in their environment.

Lester Hilton, one of our power users out of South Africa, sent me an email explaining how they use Bamboo to kick of SoapUI and JMeter scripts and how they now implemented Shift-Left with Dynatrace AppMon leveraging the plugin from Jan.

Integration Scenario with SoapUI

Lester uses SoapUI to execute Web API tests against their application. In order to integrate SoapUI tests with Dynatrace AppMon Test Automation a couple of steps are necessary. Here is a high-level overview:

Workflow of integrating Dynatrace AppMon Test Automation with SoapUI executed via Bamboo. Key step is to pass the Test Run ID from Dynatrace to the SoapUI test.

The Dynatrace Bamboo plugin will automatically register a new Test Run ID through the Register Test Session task. Because we are executing Web API Tests we chose the category Web API. The Test Run ID is stored in a variable which can be accessed via ${bamboo.dynatrace.testsession.id}. Now simply pass that variable to the script that launches SoapUI via the -P command line option. Following is a screenshot that shows these two tasks configured in a Bamboo Job:

The Dynatrace Test Run ID is passed to the SoapUI via the command line. It can then be referenced in the SoapUI script.

As the screenshot is a bit cut off – here is the full command line script to launch SoapUI:

SoapUI Script Changes

In SoapUI, Lester created a Groovy script that gets called from SoapUI while executing the tests. The script will add the X-Dynatrace HTTP Header that allows Dynatrace to identify the incoming Web Request from SoapUI. The value of that X-Dynatrace HTTP Header has to follow our Tagged Web Request specification. We require at least two values:

  • NA=Test Name -> this should be the logical name of your Test Case
  • TR=Test Run ID -> this is the Test Run ID that we received from the Bamboo Plugin

Here is the Groovy Script:

Make sure that your Load Test Script executes the Groovy script. If that is done you are all set. You can verify by doing a quick test run:

Validating that SoapUI executes all scripts correctly

That’s it. Save the Groovy project. You are all set to now execute your Bamboo Jobs. Here is what will happen

#1: The Dynatrace Plugin will register a new Test Run Id of Category Web API Test

#2: The Test Run ID is passed to the SoapUI shell that executes your script

#3: The Groovy script as part of the SoapUI project will add the X-Dynatrace header

#4: Dynatrace will capture PurePaths and understands that they come from your SoapUI script and are part of your current Bamboo Job execution

Integration Scenario with JMeter

The second integration scenario that Lester told me about was with JMeter. In his opinion JMeter gives them a bit more flexibility over SoapUI when it comes to data driven testing which is why he also integrated JMeter with Bamboo and Dynatrace AppMon.

Similar to the SoapUI scenario the first step is register a Test Run ID through the Dynatrace Bamboo Pluin. This Test Run ID now has to be passed to JMeter via the command line. JMeter provides the command line option -JNAME=VALUE to pass in a property. Here is a sample command line which executes foo.jmx, writes JMeter results into foo.jtl and passes the Dynatrace Test Run ID as property TRID:

The JMeter test result file can be fed into Bamboo’s JMeter Result Aggregator:

JMeter Script Changes

The only thing that has to be changed in your JMeter scripts is to parse the passed Test Run ID from the JMeter property. Then use this value when you create the X-dynatrace HTTP Header that you pass to every HTTP Request executed by JMeter. The NA value is going to be the Test Name and TR is going to be the Test Run ID. The easiest way to accomplish this is by using a BeanShell PreProcessor with the following code:

As you can see from the code above, the setProperty call is used to make the command line property TRID available as a regular property with the name “testrunId”.

If you want to see a live demo on how to integrate JMeter scripts with Dynatrace AppMon Test check out my YouTube Performance Clinic Tutorial on “JMeter + Dynatrace Pro Active Performance Engineering

The Final Results in Dynatrace AppMon and Bamboo

If you follow these steps you will achieve just what Lester achieved. When kicking off a build in Bamboo the Dynatrace AppMon Bamboo Plugin will first register a Test Run with Dynatrace. That Test Run ID is then passed to either the SoapUI or JMeter scripts. The scripts use this Test Run ID and pass it with the executed HTTP Requests as part of the X-Dynatrace header. This allows Dynatrace AppMon to analyze the incoming PurePaths and associate them with the correct test and Bamboo Build.

In case a code change results in a changed dynamic code behavior, e.g: a spike in SQL Executions, spike in internal API or Web Service Calls or spike in internal Exceptions Dynatrace will automatically mark these tests as degraded. With that you gain an extra level of quality metrics for your code which allows you to give faster and more accurate feedback to your developers about the impact their code changes have.

In Dynatrace AppMon you will see the quality status of all tests executed per Bamboo Build like shown here in the screenshot provided by Lester:

The Dynatrace AppMon Bamboo plugin from Jan additionally pulls this data back into Bamboo so that you can see this information in directly in Bamboo – next to all your other test results:

Thanks – And – Try it yourself

Thanks Lester for sharing your best practices when implementing real Shift-Left Quality with Bamboo, JMeter, SoapUI and Dynatrace AppMon.

If anyone wants to give this a try download the Bamboo Plugin, follow the practies in this blog, and watch some of my YouTube Tutorials where I cover other aspects of Shift-Left and Metrics-Driven Delivery Pipelines.