Running the OpenTelemetry demo application with Dynatrace

OpenTelemetry is an exciting open source observability standard that provides a common collection of tools, APIs, and SDKs to help collect observability signals. Many companies are interested in experimenting with OpenTelemetry, but have struggled with the lack of a good reference implementation.

To tackle this challenge, the OpenTelemetry community created a demo application to act as a showcase for the features and capabilities of this technology. The demo has been in active development since the summer of 2022 with Dynatrace as one of its leading contributors.

OpenTelemetry demo application boy looking through telescope
OpenTelemetry demo application hot products telescope parts

The demo application is a cloud-native e-commerce application made up of multiple microservices. Because it includes examples of 10 programming languages OpenTelemetry supports with SDKs, the application makes a good reference for developers on how to use OpenTelemetry. You can also use it to test different OpenTelemetry features and evaluate how they appear on backends. Moreover, you can use it as a framework for further customization.

OpenTelemetry demo application architecture diagram
OpenTelemetry demo application architecture diagram. Courtesy of the OpenTelemetry authors. (https://github.com/open-telemetry/opentelemetry-demo)

To run the demo, you’ll need either a Docker or Kubernetes environment. By default, the demo comes with Jaeger and Prometheus backends for displaying the collected traces and metrics, but you can easily configure alternative backends. In this example, we will be using Dynatrace.

Using Dynatrace as backend to the OpenTelemetry demo application

Dynatrace has supported OpenTelemetry since 2020. You can ingest OpenTelemetry data in two ways: using Dynatrace OneAgent technology or using OpenTelemetry Protocol (OTLP), which relies exclusively on open source technology. Both methods ingest data, but by using the Dynatrace OneAgent, users can automatically discover additional insights about their infrastructure, applications, processes, services and databases. However, in this example, we’ll deploy the demo application to send telemetry directly to Dynatrace using OTLP, so you can see how Dynatrace presents the OpenTelemetry data without the additional context OneAgent provides.

To set up the demo, follow these steps:

Step 1. Download the demo application

git clone https://github.com/open-telemetry/opentelemetry-demo.git  
cd opentelemetry-demo/

For details, see the OpenTelemetry demo application deployment documentation as a reference.

Step 2. Configure the demo application to send telemetry to Dynatrace

At the cloned repository’s root, create a new file called docker-compose.override.yml and paste the following into the file:

services: 
  otelcol: 
   environment: 
    - DT_OTLP_ENDPOINT 
    - DT_METRICS_ENDPOINT 
    - DT_API_TOKEN

This ensures all necessary environment variables are passed to all services, specifically:

  • DT_OTLP_ENDPOINT is the OTLP endpoint that is used by the collector to export traces to Dynatrace.
  • DT_METRICS_ENDPOINT is the custom metrics ingest endpoint that is used by the collector to export metrics.
  • DT_API_TOKEN is the token for your Dynatrace environment. Instructions for how to create that token follow in the next step.

We will create all of these variables in step 3.

Next, go to src/otelcollector/otelcol-config-extras.yml. This file contains a configuration that will be merged with the collector’s configuration.

Copy the following content to the file:

exporters: 
  # otlp/http exporter to export traces to Dynatrace. 
  otlphttp/dynatrace: 
   endpoint: "${DT_OTLP_ENDPOINT}" 
   headers: 
    Authorization: "Api-Token ${DT_API_TOKEN}" 
  # Dynatrace metrics exporter 
  dynatrace: 
   endpoint: "${DT_METRICS_ENDPOINT}" 
   api_token: "${DT_API_TOKEN}" 
  
processors: 
  # Processor to generate (delta temporality) metrics from 
  # spans. 
  spanmetrics/dynatrace: 
   metrics_exporter: dynatrace 
   aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA 
  
service: 
  pipelines: 
   # trace pipeline for exporting to dynatrace. 
   traces/dynatrace: 
    receivers: [otlp] 
    processors: [batch, spanmetrics/dynatrace] 
    exporters: [otlphttp/dynatrace] 
   # metrics pipeline for exporting to dynatrace. 
   metrics/dynatrace: 
    receivers: [otlp] 
    processors: [batch] 
    exporters: [dynatrace]

This configuration will export traces in the OTLP protocol using the otlp/http exporter and metrics using the Dynatrace metrics exporter. Both of these utilities are available out of the box in the OpenTelemetry collector contrib distribution.

Step 3. Set up your Dynatrace account and environment variables

Create a Dynatrace account. If you don’t have one, you can use a trial account.

Next, create an access token that includes scopes for the following:

  • Ingest metrics (metrics.ingest)
  • Ingest OpenTelemetry traces (openTelemetryTrace.ingest)

For details, see Dynatrace API – Tokens and authentication in the Dynatrace documentation.

Export the environment variables or add them to the .env file in the repository’s root. Make sure to replace the placeholder values with your token and environment ID. This example illustrates how to pass the token most easily. In an actual setup, you would configure the token using Kubernetes secrets or other secure secret handling mechanisms.

export DT_API_TOKEN=dt0c01.MY_SECRET_TOKEN
export DT_OTLP_ENDPOINT=https://{your-env-id}.live.dynatrace.com/api/v2/otlp
export DT_METRICS_ENDPOINT= https://{your-env-id}.live.dynatrace.com/api/v2/metrics/ingest
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Step 4. Deploy the application

When everything is prepared, deploy your demo application.

docker compose up --no-build

The first time running the demo, it takes a couple of minutes to download all the needed images. Afterward, the demo starts instantly, and the load generator automatically begins creating data.

You can open the demo application UI to see the app in action:

http://localhost:8080/

Use Dynatrace to observe the application

Open your Dynatrace environment in the browser (using the link from the account creation step). Using the navigation on the left, open the “Distributed traces” view. When the application is running, traces and metrics start appearing on Dynatrace. In this view, you can filter traces. For example, to visualize traces for the checkout service, you can enter “checkout” in the Filter requests field, as shown in the following image.

OpenTelemetry demo app showing distributed traces in Dynatrace

If you select one of the traces, you can see the different spans going through multiple services, as shown in the following image.

OpenTelemetry demo application HTTP post trace drilldown in Dynatrace

Using feature flags: Simulating application failures use case

One interesting use case in the OpenTelemetry Demo Application is simulating application failures. This failure logic is built into the application and it can be enabled by using a feature flag service.

To open the feature flag UI, go to:

http://localhost:8081/

You should see a feature flag called productCatalogFailure. To enable this feature flag, click edit, enable, then save.

Go to the Services page and select checkoutservice. After a minute you should start seeing some of the checkout service requests failing.

OpenTelemetry demo application checkoutservice screen in Dynatrace showing failures starting

Scroll down on the same page to see a list of traces and expand one of the failing traces, indicated by a red bar as in the picture below. In the status section of the span you can see that Dynatrace provides the reason for the failure: failed to get product #"OLJCESPC7Z.

OpenTelemetry demo application metadata tab detail in Dynatrace

What’s next

The OpenTelemetry community will continue to enhance the demo application so that it reflects the latest OpenTelemetry capabilities. Support for metrics will be extended in the near future, so stay tuned.

As a key contributor to this project, Dynatrace continues to work with the OpenTelemetry community and other vendors, to enrich its capabilities.

To learn more about Dynatrace and OpenTelemetry, join us for the on-demand Performance Clinic: Dynatrace and OpenTelemetry work #BetterTogether.