OneAgent OpenTracing and OpenTelemetry support

Early Adopter

Distributed tracing, also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture.

You can integrate OpenTracing Java, OpenTelemetry Go, and OpenTelemetry Java span data into Dynatrace. The integration is based on a standard, vendor-neutral framework for the instrumentation of your applications for distributed tracing.

Dynatrace OneAgent enables organizations to automate their OpenTracing and OpenTelemetry data collection, extend and customize PurePath® analysis with additional observability data coming from OpenTracing and/or OpenTelemetry, enrich the observability data with full-stack context, and analyze telemetry data at enterprise scale to optimize applications and drive cloud automation.

OpenTracing and/or OpenTelemetry with OneAgent enables you to:

  • Enrich monitoring data with project-specific additions (for example, custom instrumentation that adds business data or the capture of developer-specific diagnostics points).
  • Stitch together independent, unrelated transactions to extend end-to-end traces (for instance, by adding vendor-neutral custom instrumentation to gain business-process-specific or domain-specific end-to-end transactional insights).

Although we've tested the seamless integration of OpenTracing data emitted by Hazelcast and Couchbase SDK 3 with PurePath analysis, the quality of the OpenTracing spans captured by OneAgent depends on the quality of instrumentation provided by the third-party library.

Prerequisites

OpenTelemetry

OneAgent version Dynatrace version OpenTelemetry for Java version
1.215 1.216 1.0.0

OpenTracing

OneAgent version OpenTracing version
1.205 OpenTracing 0.33.0
1.219 OpenTracing 0.32.0
1.221 OpenTracing 0.31.0

Enable OpenTracing and OpenTelemetry integration

OneAgent automatically taps into traces exposed via OpenTracing and OpenTelemetry custom- or pre-instrumentation and sends the telemetry data to the Dynatrace platform.

Span settings

OneAgent version 1.215+ Dynatrace version 1.216+

Dynatrace automatically captures all OpenTracing and OpenTelemetry spans, but you can control and adapt how OpenTelemetry and OpenTracing spans are combined with OneAgent data into PurePath.

The span settings are available at Settings > Server-side service monitoring. You can define rules to:

  • Store only specific span attributes—Span attributes
  • Exclude specific spans—Span capturing
  • Define spans that should be considered as entry points—Span entry points
  • Enable context propagation for certain spans—Span context propagation

For details, see the sections that follow.

Span attributes

While Dynatrace automatically captures all OpenTracing and OpenTelemetry span attributes, it only stores span attribute values when the related keys are specified in the allowlist. By omitting span attributes containing personal data, you can meet your organization's privacy requirements and control the scope of stored monitoring data.

To add span attributes to the allowlist

  1. In the Dynatrace menu, go to Settings > Server-side service monitoring > Spans attributes
  2. Select Add item.
  3. Enter the key of a span attribute to store.
  4. Select Save changes.

Span capturing

All detected OpenTelemetry and OpenTracing spans are captured by default. This means that every detected span is added to PurePath. This gives code-level visibility along with span attributes, even for technologies not supported by OneAgent out of the box.

You can create request attributes based on spans to segment the PurePaths.

We recommend that you exclude spans for technologies supported out of the box by OneAgent for Java and Go.

To control your span capturing

  1. In the Dynatrace menu, go to Settings > Server-side service monitoring > Span capturing

  2. Select Add item.

  3. Enter a Rule name.

  4. In the Rule action list, decide whether you want to Ignore or Capture spans matching the criteria you're about to define.

  5. Select Add item.

  6. You can control span capturing based on the value of the following sources:

    • Attribute,
    • Instrumentation library name, OpenTelemetry
    • Instrumentation library version
    • Span kind—for example, server
    • Span name
  7. Select the Comparison type—for example, Contains or Equals.

  8. Enter the value for the source you specified earlier.

  9. By default, the search for key and value is not case sensitive. Turn on Case sensitive if you want your rule to consider key and value case.

  10. Select Save changes.

Span entry points

By default:

  • OpenTelemetry/OpenTracing span kinds server and consumer start new PurePaths automatically. This means that all OneAgent sensors contribute to that PurePath and a default service is created for the PurePath.
  • OpenTelemetry/OpenTracing span kinds client, internal, and producer don't start new PurePaths automatically.

You can choose to start PurePaths based on client, internal, and producer span kinds, and to opt out for server and consumer span kinds, based on various span details.

To control your span entry points

  1. Go Settings > Server-side service monitoring > Span entry points
  2. Select Add item.
  3. Enter a Rule name.
  4. In the Rule action list, specify Create entry point or Do not create entry point for spans matching the criteria you're about to define.
  5. Select Add item.
  6. You can control span entry points based on the value of the following sources
    • Span attribute
    • Instrumentation library name, OpenTelemetry
    • Instrumentation library version
    • Span kind—for example, server
    • Span name
  7. Select the Comparison type—for example, Contains or Equals.
  8. Enter the value for the source you specified earlier.
  9. By default, the search for key and value is not case sensitive. Turn on Case sensitive if you want your rule to consider key and value case.
  10. Select Save changes.

Span context propagation

Context propagation enables you to connect PurePaths through OpenTelemetry/OpenTracing. You can connect PurePath through any protocol and propagate the inject and extract usage to the Dynatrace PurePath context.

To reduce the risk of context propagation conflicts with built-in sensors, context propagation is disabled by default and is limited to spans matching the criteria of your choice.

To define rules to enable context propagation for specific spans

  1. Go Settings > Server-side service monitoring > Span context propagation
  2. Select Add item.
  3. Enter a Rule name.
  4. In the Rule action list, specify Propagate or Do not propagate span context for spans matching the criteria you're about to define.
  5. Select Add item.
  6. You can control span context propagation based on the value of the following sources
    • Span attribute
    • Instrumentation library name, OpenTelemetry
    • Instrumentation library version
    • Span kind—for example, server
    • Span name
  7. Select the Comparison type—for example, Contains or Equals.
  8. Add the value for the source you specified earlier.
  9. By default, the search for key and value is not case sensitive. Turn on Case sensitive if you want your rule to consider key and value case.
  10. Select Save changes.

Limitations

Java

OpenTracing

  • O* default service
  • When both OneAgent and OpenTracing instrumentation are present for the same technology (for example, incoming web requests via the Servlet API), you may experience the following limitations:
    • Duplicate nodes in PurePath
    • Additional overhead
    • For JDBC, such double instrumentation may break service detection Be extra cautious when enabling OneAgent OpenTracing Java support for OneAgent out-of-the-box supported technologies.
  • Integrating traces from OpenTracing Spring framework instrumentation currently is not supported.
  • The OpenTracing Java sensor doesn't capture array-type attributes.

OpenTelemetry

  • OneAgent replaces both the API-only and the SDK tracer. With OpenTelemetry Java enabled, existing tracers (such as Jaeger) will no longer see spans.
  • When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
    • Duplicate nodes in PurePath
    • Additional overhead
  • The OpenTelemetry Java sensor doesn't capture array-type attributes.

Go

  • OneAgent can only instrument Tracer implementation of the default OpenTelemetry SDK.
  • When both OneAgent and OpenTelemetry sensors are present for the same technology, you may experience the following limitations:
    • Duplicate nodes in PurePath
    • Additional overhead
  • The OpenTelemetry Go sensor doesn't capture array-type attributes.

Supported technologies

Dynatrace integrates traces from generic OpenTracing and OpenTelemetry instrumentation. We have positively tested the instrumentation of the following libraries and frameworks: