Instrument your Java application with OpenTelemetry
This walkthrough shows how to add observability to your Java application using the OpenTelemetry Java libraries and tools.
Feature | Supported |
---|---|
Automatic Instrumentation | Yes |
Automatic OneAgent Ingestion | Yes |
Prerequisites
- Dynatrace version 1.254+
- For tracing, W3C Trace Context is enabled
- From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Turn on Send W3C Trace Context HTTP headers.
Choose how to ingest data into Dynatrace
OneAgent currently only ingests traces automatically. If you are recording metrics or logs, choose the OTLP export route.
Prerequisites
- OneAgent version 1.237+
- Traces-only data
- OpenTelemetry Java Instrumentation agent support is enabled
- From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and turn on OpenTelemetry Java Instrumentation agent support.
- W3C Trace Context is enabled
- From the Dynatrace menu, go to Settings > Server-side service monitoring > Deep monitoring > Distributed tracing.
- Turn on Send W3C Trace Context HTTP headers.
Disable default exporters
Set the following two environment variables, to ensure data is provided in the right format and not exported to the default localhost
endpoint.
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER=none
Determine the API base URL
For details on how to assemble the base OTLP endpoint URL, see Export with OTLP.
The URL should end in /api/v2/otlp
.
Get API access token
The access token for ingesting traces, logs, and metrics can be generated in your Dynatrace menu under Access tokens.
Export with OTLP has more details on the format and the necessary access scopes.
Choose how you want to instrument your application
OpenTelemetry supports automatic and manual instrumentation for Java, or a combination of both.
As long as you have access to the underlying JVM and its settings, it's a good idea to start with the OpenTelemetry agent and add only manual instrumentation if the automatic approach doesn't work or doesn't provide enough information.
Even without access to the JVM, you can still manually add individual instrumentation support libraries (matching the libraries you are using in your code), which will cover a good amount of the code execution automatically.
Automatically instrument your application optional
-
Download the latest
opentelemetry-javaagent.jar
agent file and save it to a directory accessible to your application (for example,libs
). -
Configure the following environment variables to set the service and export details, substituting
[URL]
and[TOKEN]
with the values for the base URL and access token.OTEL_EXPORTER_OTLP_ENDPOINT=[URL] OTEL_EXPORTER_OTLP_HEADERS=Authorization=Api-Token [TOKEN] OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_RESOURCE_ATTRIBUTES="service.name=java-quickstart,service.version=1.0.1"
The endpoint and the token only need to be configured if you export using OTLP and do not use OneAgent ingestion.
-
Include the
-javaagent
parameter in your Java invocation command and specify the path to the agent file. For example, if you started your application from the command line:java -javaagent:/PATH/TO/opentelemetry-javaagent.jar -jar myapplication.jar
Manually instrument your application optional
Setup
-
Add the current versions of the following packages to your package configuration (e.g, Maven, Gradle).
-
Configure the following environment variables, substituting
[URL]
and[TOKEN]
with the values for the base URL and access token.OTEL_EXPORTER_OTLP_ENDPOINT=[URL] OTEL_EXPORTER_OTLP_HEADERS=Authorization=Api-Token [TOKEN] OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_RESOURCE_ATTRIBUTES="service.name=java-quickstart,service.version=1.0.1"
The endpoint and the token only need to be configured if you export using OTLP and do not use OneAgent ingestion.
-
Add the following import statements to the startup class, which bootstraps your application.
import io.opentelemetry.api.common.Attributes; import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk; import io.opentelemetry.sdk.resources.Resource;
-
Add the
initOpenTelemetry
method to your startup class and invoke it as early as possible during your application startup. This initializes OpenTelemetry for the Dynatrace backend and creates default tracer and meter providers.private static void initOpenTelemetry() { AutoConfiguredOpenTelemetrySdk.builder().addResourceCustomizer((resource, properties) -> { Resource dtMetadata = Resource.empty(); for (String name : new String[]{"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"}) { try { Properties props = new Properties(); props.load(name.startsWith("/var") ? new FileInputStream(name) : new FileInputStream(Files.readAllLines(Paths.get(name)).get(0))); dtMetadata = dtMetadata.merge(Resource.create(props.entrySet().stream() .collect(Attributes::builder, (b, e) -> b.put(e.getKey().toString(), e.getValue().toString()), (b1, b2) -> b1.putAll(b2.build())) .build()) ); } catch (IOException e) { } } return resource.merge(dtMetadata); }).build().getOpenTelemetrySdk(); }
-
Add the current versions of the following packages to your package configuration (e.g, Maven, Gradle).
-
Add the current version of opentelemetry-log4j-appender-2.17 as a runtime library to your package configuration(a
runtime
scope for Maven,runtimeOnly
for Gradle). -
Add the following import statements to the startup class, which bootstraps your application.
import io.opentelemetry.api.common.AttributeKey; import io.opentelemetry.api.common.Attributes; import io.opentelemetry.api.trace.propagation.W3CTraceContextPropagator; import io.opentelemetry.context.propagation.ContextPropagators; import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter; import io.opentelemetry.sdk.OpenTelemetrySdk; import io.opentelemetry.sdk.resources.Resource; import io.opentelemetry.sdk.trace.SdkTracerProvider; import io.opentelemetry.sdk.trace.export.BatchSpanProcessor; import io.opentelemetry.sdk.trace.export.SpanExporter; import io.opentelemetry.sdk.trace.samplers.Sampler;
-
Add two fields to your startup class for your Dynatrace URL and access token.
private static final String DT_API_URL = ""; // TODO: Provide your SaaS/Managed URL here private static final String DT_API_TOKEN = ""; // TODO: Provide the OpenTelemetry-scoped access token here
Value injectionInstead of hardcoding these values, you might also consider reading them from storage specific to your application framework (for example, environment variables or framework secrets).
-
Configure the service name using the environment variable
OTEL_SERVICE_NAME
.OTEL_SERVICE_NAME=java-quickstart
-
Add the
initOpenTelemetry
method to your initializer class and invoke it as early as possible during your application startup. This initializes OpenTelemetry for the Dynatrace backend and creates default tracer and meter providers.
private static void initOpenTelemetry()
{
// ===== GENERAL SETUP =====
// Read service name from the environment variable OTEL_SERVICE_NAME, if present
Resource serviceName = Optional.ofNullable(System.getenv("OTEL_SERVICE_NAME"))
.map(n -> Attributes.of(AttributeKey.stringKey("service.name"), n))
.map(Resource::create)
.orElseGet(Resource::empty);
// Parse the environment variable OTEL_RESOURCE_ATTRIBUTES into key-value pairs
Resource envResourceAttributes = Resource.create(Stream.of(Optional.ofNullable(System.getenv("OTEL_RESOURCE_ATTRIBUTES")).orElse("").split(","))
.filter(pair -> pair != null && pair.length() > 0 && pair.contains("="))
.map(pair -> pair.split("="))
.filter(pair -> pair.length == 2)
.collect(Attributes::builder, (b, p) -> b.put(p[0], p[1]), (b1, b2) -> b1.putAll(b2.build()))
.build()
);
// Read host information from OneAgent files to enrich telemetry
Resource dtMetadata = Resource.empty();
for (String name : new String[] {"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"}) {
try {
Properties props = new Properties();
props.load(name.startsWith("/var") ? new FileInputStream(name) : new FileInputStream(Files.readAllLines(Paths.get(name)).get(0)));
dtMetadata = dtMetadata.merge(Resource.create(
props.entrySet().stream()
.collect(Attributes::builder, (b, e) -> b.put(e.getKey().toString(), e.getValue().toString()), (b1, b2) -> b1.putAll(b2.build()))
.build()
));
} catch (IOException e) {}
}
// ===== TRACING SETUP =====
// Configure span exporter with the Dynatrace URL and the API token
SpanExporter exporter = OtlpHttpSpanExporter.builder()
.setEndpoint(DT_API_URL + "/v1/traces")
.addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
.build();
// Set up tracer provider with a batch processor and the span exporter
SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
.setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
.setSampler(Sampler.alwaysOn())
.addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
.build();
// ===== METRIC SETUP =====
// Configure metric exporter with the Dynatrace URL and the API token
OtlpHttpMetricExporter metricExporter = OtlpHttpMetricExporter.builder()
.setEndpoint(DT_API_URL + "/v1/metrics")
.addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
.setAggregationTemporalitySelector(AggregationTemporalitySelector.deltaPreferred())
.build();
// Set up meter provider with a periodic reader and the metric exporter
SdkMeterProvider meterProvider = SdkMeterProvider.builder()
.setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
.registerMetricReader(PeriodicMetricReader.builder(metricExporter).build())
.build();
// ===== LOG SETUP =====
// Configure log exporter with the Dynatrace URL and the API token
OtlpHttpLogRecordExporter logExporter = OtlpHttpLogRecordExporter.builder()
.setEndpoint(DT_API_URL + "/v1/logs")
.addHeader("Authorization", "Api-Token " + DT_API_TOKEN)
.build();
// Set up log provider with the log exporter
SdkLoggerProvider sdkLoggerProvider = SdkLoggerProvider.builder()
.setResource(Resource.getDefault().merge(envResourceAttributes).merge(serviceName).merge(dtMetadata))
.addLogProcessor(BatchLogRecordProcessor.builder(logExporter).build())
.build();
GlobalLoggerProvider.set(sdkLoggerProvider);
// ===== INITIALIZATION =====
// Initialize OpenTelemetry with the tracer and meter providers
OpenTelemetrySdk.builder()
.setTracerProvider(sdkTracerProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.setMeterProvider(meterProvider)
.setLoggerProvider(sdkLoggerProvider)
.buildAndRegisterGlobal();
//
Runtime.getRuntime().addShutdownHook(new Thread(sdkTracerProvider::close));
}
Add tracing
-
To create new spans, we first need a tracer object.
Tracer tracer = GlobalOpenTelemetry .getTracerProvider() .tracerBuilder("my-tracer") //TODO Replace with the name of your tracer .build();
-
With
tracer
, we can now use a span builder to create and start new spans.// Obtain and name new span from tracer Span span = tracer.spanBuilder("Call to /myendpoint").startSpan(); // Set demo span attributes using semantic naming span.setAttribute("http.method", "GET"); span.setAttribute("net.protocol.version", "1.1"); // Set the span as current span and parent for future child spans try (Scope scope = span.makeCurrent()) { // TODO your code goes here } finally { // Completing the span span.end(); }
In the code above, we:
- Create a new span and name it "Call to /myendpoint"
- Add two attributes, following the semantic naming convention, specific to the action of this span: information on the HTTP method and version
- Use the span's
makeCurrent()
method to mark it as active span and parent of future spans (until the span finished) - Add a
TODO
in place of the eventual business logic - Call the span's
end()
method to complete the span (in afinally
block to ensure the method is called)
Collect metrics
-
To instantiate new metrics instruments, we first need a meter object.
Meter meter = GlobalOpenTelemetry .getMeterProvider() .meterBuilder("my-meter") //TODO Replace with the name of your meter .build();
-
With
meter
, we can now create individual instruments, such as a counter.LongCounter counter = meter.counterBuilder("request_counter") .setDescription("The number of requests we received") .setUnit() .build();
-
We can now invoke the
add()
method ofcounter
to record new values with our counter and save additional attributes.Attributes attrs = Attributes.of(stringKey("ip"), "an ip address here"); counter.add(1, attrs);
You can also create an asynchronous gauge, which requires a callback function that will be invoked by OpenTelemetry upon data collection.
The following example records on each invocation the available memory, along with an attribute on the number of active users obtained from a fictitious getUserCount()
method.
meter.gaugeBuilder("free_memory")
.setDescription("Available memory in bytes")
.setUnit("bytes")
.buildWithCallback(measurement -> {
measurement.record(
Runtime.getRuntime().freeMemory(),
Attributes.of(stringKey("user_count"), getUserCount())
);
});
Connect logs
You first need to adjust your Log4j 2 configuration file log4j.xml
, to include the OpenTelemetry appender.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN" packages="io.opentelemetry.instrumentation.log4j.appender.v2_17">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout
pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} trace_id: %X{trace_id} span_id: %X{span_id} trace_flags: %X{trace_flags} - %msg%n"/>
</Console>
<OpenTelemetry name="OpenTelemetryAppender"/>
</Appenders>
<Loggers>
<Root>
<AppenderRef ref="OpenTelemetryAppender" level="All"/>
<AppenderRef ref="Console" level="All"/>
</Root>
</Loggers>
</Configuration>
In this configuration, we added a new <OpenTelemetry>
entry under <Appenders>
, as well as an <AppenderRef>
entry under <Loggers>
.
With the call to GlobalLoggerProvider
, which we previously performed under Setup, this appender is configured for the Dynatrace backend.
Ensure context propagation optional
Context propagation is particularly important when network calls (for example, REST) are involved.
If you are using automatic instrumentation and your networking libraries are covered by automatic instrumentation, this will be automatically taken care of by the instrumentation libraries. Otherwise, your code needs to take this into account.
Extracting the context when receiving a request
In the following example, we assume that we have received a network call via com.sun.net.httpserver.HttpExchange
and we define a TextMapGetter
instance to fetch the context information from the HTTP headers. We then pass that instance to extract()
, returning the context object, which allows us to continue the previous trace with our spans.
//The getter will be used for incoming requests
TextMapGetter<HttpExchange> getter =
new TextMapGetter<>() {
@Override
public String get(HttpExchange carrier, String key) {
if (carrier.getRequestHeaders().containsKey(key)) {
return carrier.getRequestHeaders().get(key).get(0);
}
return null;
}
@Override
public Iterable<String> keys(HttpExchange carrier) {
return carrier.getRequestHeaders().keySet();
}
};
public void handle(HttpExchange httpExchange) {
//Extract the SpanContext and other elements from the request
Context extractedContext = openTelemetry.getPropagators().getTextMapPropagator()
.extract(Context.current(), httpExchange, getter);
try (Scope scope = extractedContext.makeCurrent()) {
//This will automatically propagate context by creating child spans within the extracted context
Span serverSpan = tracer.spanBuilder("my-server-span") //TODO Replace with the name of your span
.setSpanKind(SpanKind.SERVER) //TODO Set the kind of your span
.startSpan();
serverSpan.setAttribute(SemanticAttributes.HTTP_METHOD, "GET"); //TODO Add attributes
serverSpan.end();
}
}
Injecting the context when sending requests
In the following example, we send a REST request to another service and provide our existing context as part of the HTTP headers of our request.
To do so, we define a TextMapSetter
instance, which adds the respective information with setRequestProperty()
. Once we have instantiated our REST object, we pass it, along with the context and the setter instance, to inject()
, which will add the necessary headers to the request.
//The setter will be used for outgoing requests
TextMapSetter<HttpURLConnection> setter =
(carrier, key, value) -> {
assert carrier != null;
// Insert the context as Header
carrier.setRequestProperty(key, value);
};
URL url = new URL("<URL>"); //TODO Replace with the URL of the service to be called
Span outGoing = tracer.spanBuilder("my-client-span") //TODO Replace with the name of your span
.setSpanKind(SpanKind.CLIENT) //TODO Set the kind of your span
.startSpan();
try (Scope scope = outGoing.makeCurrent()) {
outGoing.setAttribute(SemanticAttributes.HTTP_METHOD, "GET"); //TODO Add attributes
HttpURLConnection transportLayer = (HttpURLConnection) url.openConnection();
// Inject the request with the *current* Context, which contains our current span
openTelemetry.getPropagators().getTextMapPropagator().inject(Context.current(), transportLayer, setter);
// Make outgoing call
} finally {
outGoing.end();
}
Configure data capture to meet privacy requirements optional
While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data stored.
To view your custom span attributes, you need to allow them in the Dynatrace web UI first.
- Span attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
- Resource attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.
Verify data ingestion into Dynatrace
Once you have finished the instrumentation of your application, perform a couple of test actions to create and send demo traces, metrics, and logs and verify that they were correctly ingested into Dynatrace.
To do that for traces, in the Dynatrace menu, go to Distributed traces and select the Ingested traces tab. If you use OneAgent, select PurePaths instead.
Metrics and logs can be found under their respective entries at Observe and explore.