Instrument Rust applications with OpenTelemetry

This guide shows how to instrument your Rust application with OpenTelemetry and export the traces to Dynatrace.

To learn more about how Dynatrace works with OpenTelemetry, see Send data to Dynatrace with OpenTelemetry.

Prerequisites

Dynatrace version 1.222+
W3C Trace Context is enabled
1. From the Dynatrace menu, go to Settings > Server-side service monitoring > Deep monitoring > Distributed tracing.
2. Turn on Send W3C Trace Context HTTP headers.

Overview

To monitor your Rust application with OpenTelemetry

Instrument your application

Send the data to Dynatrace

Configure context propagation

Restart your application and verify the data in Dynatrace

Configure data capture to meet privacy requirements

Instrument your application

Rust instrumentation requires some manual steps. Add the following crates to your Cargo.toml file and add the code snippet below to any Rust method you want to monitor.
Set names for the tracer, the span, and add attributes as you see fit.

rust
opentelemetry = { version = "0.17.0", features =  ["rt-tokio"] }

rust
use opentelemetry::{global, trace::{Span, Tracer}, KeyValue};

let tracer = global::tracer("manual");
let mut span = tracer.start("my-span"); //TODO Replace with the name of your span
span.set_attribute(KeyValue::new("my-key-1", "my-value-1")); //TODO Add attributes

// your original code goes here

span.end();

Send data to Dynatrace

To send data to Dynatrace, you have to add the crates below to the Cargo.toml file, as well as add and configure the following code snippets in your Rust application code:

rust
opentelemetry-otlp = { version = "0.10.0", features = ["http-proto", "reqwest-client", "reqwest-rustls"] }
opentelemetry-http = { version =  "0.6.0" }
opentelemetry-semantic-conventions = { version = "0.9.0" }

With OneAgentWithout OneAgent

Create the following functions to initialize the tracer:

rust
use std::io::{BufRead, BufReader, Read};

use opentelemetry::{
    global,
    sdk::{propagation::TraceContextPropagator, trace as sdktrace, Resource},
    trace::{Span, TraceContextExt, TraceError, Tracer},
    Context, KeyValue,
};

use opentelemetry_http::{HeaderExtractor, HeaderInjector};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_semantic_conventions as semcov;

fn read_dt_metadata() -> Resource {
    fn read_single(path: &str, metadata: &mut Vec<KeyValue>) -> std::io::Result<()> {
        let mut file = std::fs::File::open(path)?;
        if path.starts_with("dt_metadata") {
            let mut name = String::new();
            file.read_to_string(&mut name)?;

            file = std::fs::File::open(name)?;
        }

        for line in BufReader::new(file).lines() {
            if let Some((k, v)) = line?.split_once('=') {
                metadata.push(KeyValue::new(k.to_string(), v.to_string()))
            }
        }

        Ok(())
    }

    let mut metadata = Vec::new();
    for name in [
        "dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties",
        "/var/lib/dynatrace/enrichment/dt_metadata.properties",
    ] {
        let _ = read_single(name, &mut metadata);
    }

    Resource::new(metadata)
}

fn init_tracer() -> Result<sdktrace::Tracer, TraceError> {
    global::set_text_map_propagator(TraceContextPropagator::new());


    let mut resource = Resource::new(vec![
        semcov::resource::SERVICE_NAME.string("rust-quickstart"), //TODO Replace with the name of your application
        semcov::resource::SERVICE_VERSION.string("1.0.1"), //TODO Replace with the version of your application
    ]);

    resource = resource.merge(&read_dt_metadata());

    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .http()
                .with_endpoint("http://localhost:14499/otlp/v1/traces"),
        )
        .with_trace_config(
            sdktrace::config()
                .with_resource(resource)
                .with_sampler(sdktrace::Sampler::AlwaysOn),
        )
        .install_batch(opentelemetry::runtime::Tokio)
}

When using OneAgent, make sure to enable the public Extension Execution Controller in your Dynatrace Settings, otherwise no data will be sent.

In the Dynatrace menu, go to Settings > Preferences > Extension Execution Controller. The toggles Enable Extension Execution Controller and Enable local PIPE/HTTP metric and Log Ingest API should be active.

Create the following functions to initialize the tracer:

rust
use std::collections::HashMap;
use std::io::{BufRead, BufReader, Read};

use opentelemetry::{
    global,
    sdk::{propagation::TraceContextPropagator, trace as sdktrace, Resource},
    trace::{Span, TraceContextExt, TraceError, Tracer},
    Context, KeyValue,
};

use opentelemetry_http::{HeaderExtractor, HeaderInjector};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_semantic_conventions as semcov;

fn read_dt_metadata() -> Resource {
    fn read_single(path: &str, metadata: &mut Vec<KeyValue>) -> std::io::Result<()> {
        let mut file = std::fs::File::open(path)?;
        if path.starts_with("dt_metadata") {
            let mut name = String::new();
            file.read_to_string(&mut name)?;

            file = std::fs::File::open(name)?;
        }

        for line in BufReader::new(file).lines() {
            if let Some((k, v)) = line?.split_once('=') {
                metadata.push(KeyValue::new(k.to_string(), v.to_string()))
            }
        }

        Ok(())
    }

    let mut metadata = Vec::new();
    for name in [
        "dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties",
        "/var/lib/dynatrace/enrichment/dt_metadata.properties",
    ] {
        let _ = read_single(name, &mut metadata);
    }

    Resource::new(metadata)
}

fn init_tracer() -> Result<sdktrace::Tracer, TraceError> {
    global::set_text_map_propagator(TraceContextPropagator::new());

    let mut map = HashMap::new();
    map.insert("Authorization".to_string(), "Api-Token <TOKEN>".to_string()); //TODO Replace <TOKEN> with your API Token as mentioned in the next step

    let mut resource = Resource::new(vec![
        semcov::resource::SERVICE_NAME.string("rust-quickstart"), //TODO Replace with the name of your application
        semcov::resource::SERVICE_VERSION.string("1.0.1"), //TODO Replace with the version of your application
    ]);

    resource = resource.merge(&read_dt_metadata());

    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .http()
                .with_endpoint("<URL>") //TODO Replace <URL> to your SaaS/Managed-URL as mentioned in the next step
                .with_headers(map),
        )
        .with_trace_config(
            sdktrace::config()
                .with_resource(resource)
                .with_sampler(sdktrace::Sampler::AlwaysOn),
        )
        .install_batch(opentelemetry::runtime::Tokio)
}

Lastly, you need to define the correct endpoint and token, to make sure your data arrives where it should be.

To set the endpoint:
1. Use your Environment ID to set the endpoint to which your app will send traces as follows:
  - Dynatrace SaaS https://{your-environment-id}.live.dynatrace.com/api/v2/otlp/v1/traces
  - Dynatrace Managed https://{your-domain}/e/{your-environment-id}/api/v2/otlp/v1/traces
  - Dynatrace ActiveGate https://{your-activegate-endpoint}/e/{your-environment-id}/api/v2/otlp/v1/traces
    - You may need to include the port to your ActiveGate endpoint. For example:
      - https://{your-activegate-endpoint}:9999/e/{your-environment-id}/api/v2/otlp/v1/traces
    - If you are running a containerized ActiveGate, you need to use the FQDN of it. For example:
      - https://{your-activegate-service-name}.dynatrace.svc.cluster.local/e/{your-environment-id}/api/v2/otlp/v1/traces
2. Replace <URL> in the code snippet above with your endpoint.
To create an authentication token
1. In the Dynatrace menu, go to Access tokens and select Generate new token.
2. Provide a Token name.
3. In the Search scopes box, search for Ingest OpenTelemetry traces and select the checkbox.
4. Select Generate token.
5. Select Copy to copy the token to your clipboard.
6. Save the token in a safe place; you can't display it again.
7. Replace <TOKEN> in the code snippet above with your token.

In your main method, make sure to call the function and shut it down at the end of your method:

rust
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let _ = init_tracer()?;

    //Your code goes here

    global::shutdown_tracer_provider();
}

Configure context propagation optional

If your application receives a request or calls other applications, you need to configure context propagation to make sure the spans are linked together.

Whenever receiving an incoming request, you need to extract the parent context and create the new span as a child of it. Here is how:

rust
//Method to extract the parent context from the request
fn get_parent_context(req: Request<Body>) -> Context {
    global::get_text_map_propagator(|propagator| {
        propagator.extract(&HeaderExtractor(req.headers()))
    })
}

async fn incoming_request(req: Request<Body>) -> Result<Response<Body>, Infallible> {
    let parent_cx = get_parent_context(req);

    let mut span = global::tracer("manual-server")
        .start_with_context("my-server-span", &parent_cx); //TODO Replace with the name of your span

    span.set_attribute(KeyValue::new("my-server-key-1", "my-server-value-1")); //TODO Add attributes

    //Your code goes here
}

If your application calls another service, you need to ensure that you propagate the context, adding it to your outgoing request. Here is how:

rust
async fn outgoing_request(
    context: Context,
) -> std::result::Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    let client = Client::new();
    let span = global::tracer("manual-client")
        .start_with_context("my-client-span", &context); //TODO Replace with the name of your span
    let cx = Context::current_with_span(span);

    let mut req = hyper::Request::builder().uri("<HTTP_URL>");

    //Method to inject the current context in the request
    global::get_text_map_propagator(|propagator| {
        propagator.inject_context(&cx, &mut HeaderInjector(&mut req.headers_mut().unwrap()))
    });

    cx.span()
        .set_attribute(KeyValue::new("my-client-key-1", "my-client-value-1")); //TODO Add attributes

    //Your code goes here
}

Verify that the traces are ingested into Dynatrace

A few minutes after restarting your app, look for your spans:

In the Dynatrace menu, go to Distributed traces and select the Ingested traces tab.
- If you use OneAgent, go to Distributed traces and select the PurePaths tab.
Your spans will be part of an existing PurePath, if the root of your call is already monitored by OneAgent.

If your application does not receive any traffic, there will be no traces.

Configure data capture to meet privacy requirements optional

While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data that's stored.

To view your custom span attributes, you need to allow them in the Dynatrace web UI first:

Span attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
Resource attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.