Instrument Rust applications with OpenTelemetry
This guide shows how to instrument your Rust application with OpenTelemetry and export the traces to Dynatrace.
To learn more about how Dynatrace works with OpenTelemetry, see Send data to Dynatrace with OpenTelemetry.
Prerequisites
- Dynatrace version 1.222+
- W3C Trace Context is enabled
- From the Dynatrace menu, go to Settings > Server-side service monitoring > Deep monitoring > Distributed tracing.
- Turn on Send W3C Trace Context HTTP headers.
Overview
To monitor your Rust application with OpenTelemetry
Instrument your application
Send the data to Dynatrace
Configure context propagation
Restart your application and verify the data in Dynatrace
Configure data capture to meet privacy requirements
Instrument your application
Rust instrumentation requires some manual steps. Add the following crates to your Cargo.toml
file and add the code snippet below to any Rust method you want to monitor.
Set names for the tracer, the span, and add attributes as you see fit.
opentelemetry = { version = "0.17.0", features = ["rt-tokio"] }
use opentelemetry::{global, trace::{Span, Tracer}, KeyValue};
let tracer = global::tracer("manual");
let mut span = tracer.start("my-span"); //TODO Replace with the name of your span
span.set_attribute(KeyValue::new("my-key-1", "my-value-1")); //TODO Add attributes
// your original code goes here
span.end();
Send data to Dynatrace
To send data to Dynatrace, you have to add the crates below to the Cargo.toml
file, as well as add and configure the following code snippets in your Rust application code:
opentelemetry-otlp = { version = "0.10.0", features = ["http-proto", "reqwest-client", "reqwest-rustls"] }
opentelemetry-http = { version = "0.6.0" }
opentelemetry-semantic-conventions = { version = "0.9.0" }
- Create the following functions to initialize the tracer:
use std::io::{BufRead, BufReader, Read};
use opentelemetry::{
global,
sdk::{propagation::TraceContextPropagator, trace as sdktrace, Resource},
trace::{Span, TraceContextExt, TraceError, Tracer},
Context, KeyValue,
};
use opentelemetry_http::{HeaderExtractor, HeaderInjector};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_semantic_conventions as semcov;
fn read_dt_metadata() -> Resource {
fn read_single(path: &str, metadata: &mut Vec<KeyValue>) -> std::io::Result<()> {
let mut file = std::fs::File::open(path)?;
if path.starts_with("dt_metadata") {
let mut name = String::new();
file.read_to_string(&mut name)?;
file = std::fs::File::open(name)?;
}
for line in BufReader::new(file).lines() {
if let Some((k, v)) = line?.split_once('=') {
metadata.push(KeyValue::new(k.to_string(), v.to_string()))
}
}
Ok(())
}
let mut metadata = Vec::new();
for name in [
"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties",
"/var/lib/dynatrace/enrichment/dt_metadata.properties",
] {
let _ = read_single(name, &mut metadata);
}
Resource::new(metadata)
}
fn init_tracer() -> Result<sdktrace::Tracer, TraceError> {
global::set_text_map_propagator(TraceContextPropagator::new());
let mut resource = Resource::new(vec![
semcov::resource::SERVICE_NAME.string("rust-quickstart"), //TODO Replace with the name of your application
semcov::resource::SERVICE_VERSION.string("1.0.1"), //TODO Replace with the version of your application
]);
resource = resource.merge(&read_dt_metadata());
opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(
opentelemetry_otlp::new_exporter()
.http()
.with_endpoint("http://localhost:14499/otlp/v1/traces"),
)
.with_trace_config(
sdktrace::config()
.with_resource(resource)
.with_sampler(sdktrace::Sampler::AlwaysOn),
)
.install_batch(opentelemetry::runtime::Tokio)
}
When using OneAgent, make sure to enable the public Extension Execution Controller in your Dynatrace Settings, otherwise no data will be sent.
In the Dynatrace menu, go to Settings > Preferences > Extension Execution Controller. The toggles Enable Extension Execution Controller and Enable local PIPE/HTTP metric and Log Ingest API should be active.
- Create the following functions to initialize the tracer:
use std::collections::HashMap;
use std::io::{BufRead, BufReader, Read};
use opentelemetry::{
global,
sdk::{propagation::TraceContextPropagator, trace as sdktrace, Resource},
trace::{Span, TraceContextExt, TraceError, Tracer},
Context, KeyValue,
};
use opentelemetry_http::{HeaderExtractor, HeaderInjector};
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_semantic_conventions as semcov;
fn read_dt_metadata() -> Resource {
fn read_single(path: &str, metadata: &mut Vec<KeyValue>) -> std::io::Result<()> {
let mut file = std::fs::File::open(path)?;
if path.starts_with("dt_metadata") {
let mut name = String::new();
file.read_to_string(&mut name)?;
file = std::fs::File::open(name)?;
}
for line in BufReader::new(file).lines() {
if let Some((k, v)) = line?.split_once('=') {
metadata.push(KeyValue::new(k.to_string(), v.to_string()))
}
}
Ok(())
}
let mut metadata = Vec::new();
for name in [
"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties",
"/var/lib/dynatrace/enrichment/dt_metadata.properties",
] {
let _ = read_single(name, &mut metadata);
}
Resource::new(metadata)
}
fn init_tracer() -> Result<sdktrace::Tracer, TraceError> {
global::set_text_map_propagator(TraceContextPropagator::new());
let mut map = HashMap::new();
map.insert("Authorization".to_string(), "Api-Token <TOKEN>".to_string()); //TODO Replace <TOKEN> with your API Token as mentioned in the next step
let mut resource = Resource::new(vec![
semcov::resource::SERVICE_NAME.string("rust-quickstart"), //TODO Replace with the name of your application
semcov::resource::SERVICE_VERSION.string("1.0.1"), //TODO Replace with the version of your application
]);
resource = resource.merge(&read_dt_metadata());
opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(
opentelemetry_otlp::new_exporter()
.http()
.with_endpoint("<URL>") //TODO Replace <URL> to your SaaS/Managed-URL as mentioned in the next step
.with_headers(map),
)
.with_trace_config(
sdktrace::config()
.with_resource(resource)
.with_sampler(sdktrace::Sampler::AlwaysOn),
)
.install_batch(opentelemetry::runtime::Tokio)
}
Lastly, you need to define the correct endpoint and token, to make sure your data arrives where it should be.
- To set the endpoint:
- Use your Environment ID to set the endpoint to which your app will send traces as follows:
- Dynatrace SaaS
https://{your-environment-id}.live.dynatrace.com/api/v2/otlp/v1/traces
- Dynatrace Managed
https://{your-domain}/e/{your-environment-id}/api/v2/otlp/v1/traces
- Dynatrace ActiveGate
https://{your-activegate-endpoint}/e/{your-environment-id}/api/v2/otlp/v1/traces
- You may need to include the port to your ActiveGate endpoint. For example:
https://{your-activegate-endpoint}:9999/e/{your-environment-id}/api/v2/otlp/v1/traces
- If you are running a containerized ActiveGate, you need to use the FQDN of it. For example:
https://{your-activegate-service-name}.dynatrace.svc.cluster.local/e/{your-environment-id}/api/v2/otlp/v1/traces
- You may need to include the port to your ActiveGate endpoint. For example:
- Dynatrace SaaS
- Replace
<URL>
in the code snippet above with your endpoint.
- Use your Environment ID to set the endpoint to which your app will send traces as follows:
- To create an authentication token
- In the Dynatrace menu, go to Access tokens and select Generate new token.
- Provide a Token name.
- In the Search scopes box, search for
Ingest OpenTelemetry traces
and select the checkbox. - Select Generate token.
- Select Copy to copy the token to your clipboard.
- Save the token in a safe place; you can't display it again.
- Replace
<TOKEN>
in the code snippet above with your token.
- In your main method, make sure to call the function and shut it down at the end of your method:
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let _ = init_tracer()?;
//Your code goes here
global::shutdown_tracer_provider();
}
Configure context propagation optional
If your application receives a request or calls other applications, you need to configure context propagation to make sure the spans are linked together.
- Whenever receiving an incoming request, you need to extract the parent context and create the new span as a child of it. Here is how:
//Method to extract the parent context from the request
fn get_parent_context(req: Request<Body>) -> Context {
global::get_text_map_propagator(|propagator| {
propagator.extract(&HeaderExtractor(req.headers()))
})
}
async fn incoming_request(req: Request<Body>) -> Result<Response<Body>, Infallible> {
let parent_cx = get_parent_context(req);
let mut span = global::tracer("manual-server")
.start_with_context("my-server-span", &parent_cx); //TODO Replace with the name of your span
span.set_attribute(KeyValue::new("my-server-key-1", "my-server-value-1")); //TODO Add attributes
//Your code goes here
}
- If your application calls another service, you need to ensure that you propagate the context, adding it to your outgoing request. Here is how:
async fn outgoing_request(
context: Context,
) -> std::result::Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
let client = Client::new();
let span = global::tracer("manual-client")
.start_with_context("my-client-span", &context); //TODO Replace with the name of your span
let cx = Context::current_with_span(span);
let mut req = hyper::Request::builder().uri("<HTTP_URL>");
//Method to inject the current context in the request
global::get_text_map_propagator(|propagator| {
propagator.inject_context(&cx, &mut HeaderInjector(&mut req.headers_mut().unwrap()))
});
cx.span()
.set_attribute(KeyValue::new("my-client-key-1", "my-client-value-1")); //TODO Add attributes
//Your code goes here
}
Verify that the traces are ingested into Dynatrace
A few minutes after restarting your app, look for your spans:
- In the Dynatrace menu, go to Distributed traces and select the Ingested traces tab.
- If you use OneAgent, go to Distributed traces and select the PurePaths tab.
- Your spans will be part of an existing PurePath, if the root of your call is already monitored by OneAgent.
If your application does not receive any traffic, there will be no traces.
Configure data capture to meet privacy requirements optional
While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data that's stored.
To view your custom span attributes, you need to allow them in the Dynatrace web UI first:
- Span attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
- Resource attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.