Instrument your Go application with OpenTelemetry
This walkthrough shows how to add observability to your Go application using the OpenTelemetry Go libraries and tools.
Feature | Supported |
---|---|
Automatic Instrumentation | Yes |
Automatic OneAgent Ingestion | Yes |
Prerequisites
- Dynatrace version 1.222+
- For tracing, W3C Trace Context is enabled
- From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Turn on Send W3C Trace Context HTTP headers.
Choose how to ingest data into Dynatrace
OneAgent currently only ingests traces automatically. If you are recording metrics or logs, choose the OTLP export route.
Apart from the following prerequisites, there are no other steps necessary when using OneAgent to ingest Go OpenTelemetry data.
Prerequisites
- OneAgent version 1.207+
- Traces-only data
- OpenTelemetry Go Instrumentation agent support is enabled
- From the Dynatrace menu, go to Settings > Preferences > OneAgent features.
- Find and turn on OpenTelemetry (Go).
Determine the API base URL
For details on how to assemble the base OTLP endpoint URL, see Export with OTLP. The URL should end in /api/v2/otlp
.
Get API access token
The access token for ingesting traces, logs, and metrics can be generated in your Dynatrace menu under Access tokens.
Export with OTLP has more details on the format and the necessary access scopes.
Choose how you want to instrument your application
OpenTelemetry supports on Go automatic and manual instrumentation, or a combination of both.
It's a good idea to start with automatic instrumentation and add manual instrumentation if the automatic approach doesn't work or doesn't provide enough information.
Initialize OpenTelemetry
The initialization of OpenTelemetry varies, depending on whether you use OneAgent to send data to Dynatrace or perform an OTLP export.
-
Run the following command to install the OpenTelemetry SDK.
go get go.opentelemetry.io/otel/sdk
-
Add the following import statements.
import ( "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/propagation" sdktrace "go.opentelemetry.io/otel/sdk/trace" )
-
Add the following code to your
main
function, to initialize OpenTelemetry.otel.SetTracerProvider(sdktrace.NewTracerProvider()) otel.SetTextMapPropagator(propagation.TraceContext{})
-
Run the following commands to install all necessary dependencies.
go get github.com/Dynatrace/OneAgent-SDK-for-Go go get go.opentelemetry.io/otel go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp go get go.opentelemetry.io/otel/sdk
-
Add the following import statements.
import ( "context" "github.com/Dynatrace/OneAgent-SDK-for-Go/sdk" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/propagation" sdkmetric "go.opentelemetry.io/otel/sdk/metric" "go.opentelemetry.io/otel/sdk/metric/metricdata" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.20.0" "log" "time" )
-
Add the following code to your startup file and provide the right values for
DT_API_HOST
andDT_API_TOKEN
.DT_API_HOST
should contain only the hostname of your Dynatrace URL (for example,XXXXX.live.dyntrace.com
); it is not a URL and must not contain any schemas or pathsDT_API_TOKEN
should contain the access token
func InitOpenTelemetry() { // ===== GENERAL SETUP ===== DT_API_HOST := "" // Only the host part of your Dynatrace URL DT_API_BASE_PATH := "/api/v2/otlp" DT_API_TOKEN := "" authHeader := map[string]string{"Authorization": "Api-Token " + DT_API_TOKEN} ctx := context.Background() oneagentsdk := sdk.CreateInstance() dtMetadata := oneagentsdk.GetEnrichmentMetadata() var attributes []attribute.KeyValue for k, v := range dtMetadata { attributes = append(attributes, attribute.KeyValue{Key: attribute.Key(k), Value: attribute.StringValue(v)}) } attributes = append(attributes, semconv.ServiceNameKey.String("go-quickstart"), //TODO Replace with the name of your application semconv.ServiceVersionKey.String("1.0.1"), //TODO Replace with the version of your application ) res, err := resource.New(ctx, resource.WithAttributes(attributes...)) if err != nil { log.Fatalf("Failed to create resource: %v", err) } // ===== TRACING SETUP ===== exporter, err := otlptracehttp.New( ctx, otlptracehttp.WithEndpoint(DT_API_HOST), otlptracehttp.WithURLPath(DT_API_BASE_PATH+"/v1/traces"), otlptracehttp.WithHeaders(authHeader), ) if err != nil { log.Fatalf("Failed to create OTLP exporter: %v", err) } tp := sdktrace.NewTracerProvider( sdktrace.WithResource(res), sdktrace.WithSampler(sdktrace.AlwaysSample()), sdktrace.WithBatcher(exporter), ) otel.SetTracerProvider(tp) otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{})) // ===== METRIC SETUP ===== var deltaTemporalitySelector = func(sdkmetric.InstrumentKind) metricdata.Temporality { return metricdata.DeltaTemporality } metricsExporter, err := otlpmetrichttp.New( ctx, otlpmetrichttp.WithEndpoint(DT_API_HOST), otlpmetrichttp.WithURLPath(DT_API_BASE_PATH+"/v1/metrics"), otlpmetrichttp.WithHeaders(authHeader), otlpmetrichttp.WithTemporalitySelector(deltaTemporalitySelector), ) if err != nil { log.Fatalf("Failed to create OTLP exporter: %v", err) } mp := sdkmetric.NewMeterProvider( sdkmetric.WithResource(res), sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricsExporter, sdkmetric.WithInterval(2*time.Second))), ) otel.SetMeterProvider(mp) }
-
Make sure to call
InitOpenTelemetry
as early as possible in your startup code to initialize OpenTelemetry.
Automatically instrument your application optional
-
Browse the OpenTelemetry registry and pick the instrumentation libraries matching your application libraries.
-
Install the applicable support libraries using
go get
.go get go.opentelemetry.io/contrib/instrumentation/[PACKAGE_DETAILS]
-
Add the package to your import statements.
import ( "go.opentelemetry.io/[PACKAGE]" )
-
Wrap your existing code with calls to the support libraries.
Example for net/http
-
Install the instrumentation library for
net/http
. -
Add the package to your import statements.
import ( // other packages "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" )
-
Wrap your HTTP handler function.
handler := http.HandlerFunc(httpHandler) wrappedHandler := otelhttp.NewHandler(handler, "my-span") //TODO Replace with the name of your span //Use the wrappedHandler with your handle http.Handle("/", wrappedHandler)
Manually instrument your application
Add tracing
-
You first need to get a tracer object.
tracer := otel.Tracer("my-tracer")
-
With
tracer
, you can now use a span builder to create and start new spans._, span := tracer.Start(r.Context(), "Call to /myendpoint") defer span.End() span.SetAttributes(attribute.String("http.method", "GET"), attribute.String("net.protocol.version", "1.1")) // TODO your code goes here
In the code above, we:
- Create a new span and name it "Call to /myendpoint"
- Schedule a deferred call to
End()
, to ensure the span is properly closed when the function returns - Add two attributes, following the semantic naming convention, specific to the action of this span: information on the HTTP method and version
- Add a
TODO
in place of the eventual business logic
Collect metrics
-
Obtain a meter object.
meter := otel.Meter("my-meter")
-
With
meter
, we can now create individual instruments, such as a counter.requestCounter, _ := meter.Int64Counter("request_counter")
-
Now we can invoke the
Add()
method ofrequestCounter
to record new values with the counter.requestCounter.Add(context.Background(), 1)
Connect logs
OpenTelemetry logging is currently not yet available for Go and is still under development.
Ensure context propagation optional
Context propagation is particularly important when network calls (for example, REST) are involved.
Extracting the context when receiving a request
In the following example, we assume that we have received a network call via the net/http
library and its Request
type.
To obtain a handle to the original context (which was provided by the calling service), we pass the HTTP header object (r.Header
) to the Extract
function of the global propagator singleton, which instantiates that context and returns in parentCtx
. This allows us to continue the previous trace with our own spans.
func httpHandler(w http.ResponseWriter, r *http.Request) {
parentCtx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
ctx, span := tracer.Start(
parentCtx,
"manual-server", //TODO Replace with the name of your span
trace.WithAttributes(
attribute.String("my-key-1", "my-value-1") //TODO Add attributes
)
)
defer span.End()
//TODO your code goes here
}
Injecting the context when sending requests
In the following example, we set up a new instance of Request
and pass the object to the Inject
call of the global propagator singleton. This adds the necessary HTTP headers to the request object, which we eventually pass to Do
to execute the network request.
client := http.Client{}
req, err := http.NewRequest("<method>", "<url>", <body>)
if err != nil {
// TODO handle error
}
//Method to inject the current context in the request headers
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
client.Do(req) // Your call goes here
Configure data capture to meet privacy requirements optional
While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data stored.
To view your custom span attributes, you need to allow them in the Dynatrace web UI first.
- Span attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
- Resource attributes: In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.
Verify data ingestion into Dynatrace
Once you have finished the instrumentation of your application, perform a couple of test actions to create and send demo traces, metrics, and logs and verify that they were correctly ingested into Dynatrace.
To do that for traces, in the Dynatrace menu, go to Distributed traces and select the Ingested traces tab. If you use OneAgent, select PurePaths instead.
Metrics and logs can be found under their respective entries at Observe and explore.