Instrument your Java application with OpenTelemetry

This walkthrough shows how to add observability to your Java application using the OpenTelemetry Java libraries and tools.

Prerequisites

• Dynatrace version 1.254+
• For tracing, W3C Trace Context is enabled
  1. From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
  2. Turn on Send W3C Trace Context HTTP headers.

Choose how to ingest data into Dynatrace

plaintextcopydownload
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER=none

Choose how you want to instrument your application

OpenTelemetry supports automatic and manual instrumentation for Java, or a combination of both.

Automatically instrument your application optional

1. Download the latest opentelemetry-javaagent.jar agent file and save it to a directory accessible to your application (for example, libs).

2. Configure the following environment variables to set the service and export details, substituting [URL] and [TOKEN] with the values for the base URL and access token.

   shellcopydownload
   OTEL_EXPORTER_OTLP_ENDPOINT=[URL]
   OTEL_EXPORTER_OTLP_HEADERS=Authorization=Api-Token [TOKEN]
   OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
   OTEL_RESOURCE_ATTRIBUTES="service.name=java-quickstart,service.version=1.0.1"

   The endpoint and the token only need to be configured if you export using OTLP and do not use OneAgent ingestion.

3. Include the -javaagent parameter in your Java invocation command and specify the path to the agent file. For example, if you started your application from the command line:

   shellcopydownload
   java -javaagent:/PATH/TO/opentelemetry-javaagent.jar -jar myapplication.jar

Manually instrument your application optional

Setup

javacopydownload
private static void initOpenTelemetry()
{
    // ===== GENERAL SETUP =====

    // Read service name from the environment variable OTEL_SERVICE_NAME, if present
    Resource serviceName = Optional.ofNullable(System.getenv("OTEL_SERVICE_NAME"))
        .map(n -> Attributes.of(AttributeKey.stringKey("service.name"), n))
        .map(Resource::create)
        .orElseGet(Resource::empty);

    // Parse the environment variable OTEL_RESOURCE_ATTRIBUTES into key-value pairs
    Resource envResourceAttributes = Resource.create(Stream.of(Optional.ofNullable(System.getenv("OTEL_RESOURCE_ATTRIBUTES")).orElse("").split(","))
        .filter(pair -> pair != null && pair.length() > 0 && pair.contains("="))
        .map(pair -> pair.split("="))
        .filter(pair -> pair.length == 2)
        .collect(Attributes::builder, (b, p) -> b.put(p[0], p[1]), (b1, b2) -> b1.putAll(b2.build()))
        .build()
    );

    // Read host information from OneAgent files to enrich telemetry
    Resource dtMetadata = Resource.empty();
    for (String name : new String[] {"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"}) {
        try {
            Properties props = new Properties();
            props.load(name.startsWith("/var") ? new FileInputStream(name) : new FileInputStream(Files.readAllLines(Paths.get(name)).get(0)));

            dtMetadata = dtMetadata.merge(Resource.create(
                props.entrySet().stream()
                    .collect(Attributes::builder, (b, e) -> b.put(e.getKey().toString(), e.getValue().toString()), (b1, b2) -> b1.putAll(b2.build()))
                    .build()
            ));
        } catch (IOException e) {}
    }


    // ===== TRACING SETUP =====

    // Configure span exporter with the Dynatrace URL and the API token
    SpanExporter exporter = OtlpHttpSpanExporter.builder()
        .setEndpoint(DT_API_URL + "/v1/traces")
        .addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
        .build();

    // Set up tracer provider with a batch processor and the span exporter
    SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
        .setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
        .setSampler(Sampler.alwaysOn())
        .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
        .build();


    // ===== METRIC SETUP =====

    // Configure metric exporter with the Dynatrace URL and the API token
    OtlpHttpMetricExporter metricExporter = OtlpHttpMetricExporter.builder()
        .setEndpoint(DT_API_URL + "/v1/metrics")
        .addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
        .setAggregationTemporalitySelector(AggregationTemporalitySelector.deltaPreferred())
        .build();

    // Set up meter provider with a periodic reader and the metric exporter
    SdkMeterProvider meterProvider = SdkMeterProvider.builder()
        .setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
        .registerMetricReader(PeriodicMetricReader.builder(metricExporter).build())
        .build();


    // ===== LOG SETUP =====

    // Configure log exporter with the Dynatrace URL and the API token
    OtlpHttpLogRecordExporter logExporter = OtlpHttpLogRecordExporter.builder()
        .setEndpoint(DT_API_URL + "/v1/logs")
        .addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
        .build();

    // Set up log provider with the log exporter
    SdkLoggerProvider sdkLoggerProvider = SdkLoggerProvider.builder()
        .setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
        .addLogProcessor(BatchLogRecordProcessor.builder(logExporter).build())
        .build();

    GlobalLoggerProvider.set(sdkLoggerProvider);


    // ===== INITIALIZATION =====

    // Initialize OpenTelemetry with the tracer and meter providers
    OpenTelemetrySdk.builder()
        .setTracerProvider(sdkTracerProvider)
        .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
        .setMeterProvider(meterProvider)
        .setLoggerProvider(sdkLoggerProvider)
        .buildAndRegisterGlobal();

    // 
    Runtime.getRuntime().addShutdownHook(new Thread(sdkTracerProvider::close));
}

Add tracing

1. To create new spans, we first need a tracer object.

   javacopydownload
   Tracer tracer = GlobalOpenTelemetry
       .getTracerProvider()
       .tracerBuilder("my-tracer") //TODO Replace with the name of your tracer
       .build();

2. With tracer, we can now use a span builder to create and start new spans.

   javacopydownload
   // Obtain and name new span from tracer
   Span span = tracer.spanBuilder("Call to /myendpoint").startSpan();
   
   // Set demo span attributes using semantic naming
   span.setAttribute("http.method", "GET");
   span.setAttribute("net.protocol.version", "1.1");
   
   // Set the span as current span and parent for future child spans
   try (Scope scope = span.makeCurrent())
   {
       // TODO your code goes here
   }
   finally
   {
       // Completing the span
       span.end();
   }

   In the code above, we:

   • Create a new span and name it "Call to /myendpoint"
   • Add two attributes, following the semantic naming convention, specific to the action of this span: information on the HTTP method and version
   • Use the span's makeCurrent() method to mark it as active span and parent of future spans (until the span finished)
   • Add a TODO in place of the eventual business logic
   • Call the span's end() method to complete the span (in a finally block to ensure the method is called)

Collect metrics

1. To instantiate new metrics instruments, we first need a meter object.

   javacopydownload
   Meter meter = GlobalOpenTelemetry
       .getMeterProvider()
       .meterBuilder("my-meter") //TODO Replace with the name of your meter
       .build();

2. With meter, we can now create individual instruments, such as a counter.

   javacopydownload
   LongCounter counter = meter.counterBuilder("request_counter")
       .setDescription("The number of requests we received")
       .setUnit()
       .build();

3. We can now invoke the add() method of counter to record new values with our counter and save additional attributes.

   javacopydownload
   Attributes attrs = Attributes.of(stringKey("ip"), "an ip address here");
   
   counter.add(1, attrs);

You can also create an asynchronous gauge, which requires a callback function that will be invoked by OpenTelemetry upon data collection.

The following example records on each invocation the available memory, along with an attribute on the number of active users obtained from a fictitious getUserCount() method.

javacopydownload
meter.gaugeBuilder("free_memory")
    .setDescription("Available memory in bytes")
    .setUnit("bytes")
    .buildWithCallback(measurement -> {
        measurement.record(
            Runtime.getRuntime().freeMemory(),
            Attributes.of(stringKey("user_count"), getUserCount())
        );
    });

Connect logs

You first need to adjust your Log4j 2 configuration file log4j.xml, to include the OpenTelemetry appender.

xmlcopydownload
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout
          pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
    </Console>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root>
      <AppenderRef ref="OpenTelemetryAppender" level="All"/>
      <AppenderRef ref="Console" level="All"/>
    </Root>
  </Loggers>
</Configuration>

In this configuration, we added a new <OpenTelemetry> entry under <Appenders>, as well as an <AppenderRef> entry under <Loggers>.

With the call to GlobalLoggerProvider, which we previously performed under Setup, this appender is configured for the Dynatrace backend.

Ensure context propagation optional

Context propagation is particularly important when network calls (for example, REST) are involved.

If you are using automatic instrumentation and your networking libraries are covered by automatic instrumentation, this will be automatically taken care of by the instrumentation libraries. Otherwise, your code needs to take this into account.

Extracting the context when receiving a request

In the following example, we assume that we have received a network call via com.sun.net.httpserver.HttpExchange and we define a TextMapGetter instance to fetch the context information from the HTTP headers. We then pass that instance to extract(), returning the context object, which allows us to continue the previous trace with our spans.

javacopydownload
//The getter will be used for incoming requests
TextMapGetter<HttpExchange> getter =
    new TextMapGetter<>() {
        @Override
        public String get(HttpExchange carrier, String key) {
            if (carrier.getRequestHeaders().containsKey(key)) {
                return carrier.getRequestHeaders().get(key).get(0);
            }
            return null;
        }

        @Override
        public Iterable<String> keys(HttpExchange carrier) {
            return carrier.getRequestHeaders().keySet();
        }
    };

public void handle(HttpExchange httpExchange) {
    //Extract the SpanContext and other elements from the request
    Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
        .extract(Context.current(), httpExchange, getter);
    try (Scope scope = extractedContext.makeCurrent()) {
        //This will automatically propagate context by creating child spans within the extracted context
        Span serverSpan = tracer.spanBuilder("my-server-span") //TODO Replace with the name of your span
            .setSpanKind(SpanKind.SERVER) //TODO Set the kind of your span
            .startSpan();
        serverSpan.setAttribute(SemanticAttributes.HTTP_METHOD, "GET"); //TODO Add attributes
        serverSpan.end();
    }
}

Injecting the context when sending requests

In the following example, we send a REST request to another service and provide our existing context as part of the HTTP headers of our request.

To do so, we define a TextMapSetter instance, which adds the respective information with setRequestProperty(). Once we have instantiated our REST object, we pass it, along with the context and the setter instance, to inject(), which will add the necessary headers to the request.

javacopydownload
//The setter will be used for outgoing requests
TextMapSetter<HttpURLConnection> setter =
    (carrier, key, value) -> {
        assert carrier != null;
        // Insert the context as Header
        carrier.setRequestProperty(key, value);
    };

URL url = new URL("<URL>"); //TODO Replace with the URL of the service to be called
Span outGoing = tracer.spanBuilder("my-client-span") //TODO Replace with the name of your span
    .setSpanKind(SpanKind.CLIENT) //TODO Set the kind of your span
    .startSpan();
try (Scope scope = outGoing.makeCurrent()) {
    outGoing.setAttribute(SemanticAttributes.HTTP_METHOD, "GET"); //TODO Add attributes
    HttpURLConnection transportLayer = (HttpURLConnection) url.openConnection();
    // Inject the request with the *current*  Context, which contains our current span
    openTelemetry.getPropagators().getTextMapPropagator().inject(Context.current(), transportLayer, setter);
    // Make outgoing call
} finally {
    outGoing.end();
}

Configure data capture to meet privacy requirements optional

While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data stored.

To view your custom span attributes, you need to allow them in the Dynatrace web UI first.

• Span attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
• Resource attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.

Verify data ingestion into Dynatrace

Once you have finished the instrumentation of your application, perform a couple of test actions to create and send demo traces, metrics, and logs and verify that they were correctly ingested into Dynatrace.

To do that for traces, in the Dynatrace menu, go to Distributed traces and select the Ingested traces tab. If you use OneAgent, select PurePaths instead.

Metrics and logs can be found under their respective entries at Observe and explore.