Instrument C++ applications with OpenTelemetry

This guide shows how to instrument your C++ application with OpenTelemetry and export the traces to Dynatrace.

To learn more about how Dynatrace works with OpenTelemetry, see Send data to Dynatrace with OpenTelemetry.

Prerequisites

Dynatrace version 1.222+
W3C Trace Context is enabled
1. From the Dynatrace menu, go to Settings > Server-side service monitoring > Deep monitoring > Distributed tracing.
2. Turn on Send W3C Trace Context HTTP headers.

OpenTelemetry-cpp prerequisites

A supported development platform.
A compatible C++ compiler.
Git for fetching opentelemetry-cpp source code from repository. For details, see how to set up git.
CMake for building opentelemetry-cpp API and SDK. For details, see how to install CMake.
GoogleTest framework to build and run the OpenTelemetry unit tests. For details, see how to install GoogleTest.
Google benchmark, a library to benchmark code snippets. For details, see how to install benchmark.

OpenTelemetry-cpp OTLP exporter prerequisites

Protobuf library to serialize structured data. For details, see how to install Protobuf.
libcurl the multiprotocol file transfer library. For details, see (libcurl documentation).
nlohmann/json the JSON for Modern C++ (you can use cmake to build and install nlohmann/json from its github repository).

Build and install the OpenTelemetry-cpp

Now that you have all the dependencies, let's build the opentelemetry-cpp using CMake.
We will follow the steps from OpenTelemetry, with some adjustments to also build the OTLP_HTTP exporter.

Get the opentelemetry-cpp source in the directory where you want to create the code repository:

shell
git clone --recursive https://github.com/open-telemetry/opentelemetry-cpp

Navigate to the cloned repository and create the CMake build configuration:

shell
cd opentelemetry-cpp
mkdir build && cd build
cmake -DWITH_OTLP=ON -DWITH_OTLP_HTTP=ON ..

Once the build configuration is created, build the CMake targets. This step includes building SDKs and the unit tests for API and SDK. Note that, since the API is a header-only library, no separate build is triggered for it.

shell
cmake --build . --target all

To validate your build, run the tests:

shell
ctest

Install the header files for the API, and the generated targets and header files for the SDK. You can also define the install location by adding --prefix /<install_root>/ to the command below, or execute it as is to use the default one.

shell
cmake --install .

Overview

To monitor your C++ application with OpenTelemetry

Instrument your application

Send the data to Dynatrace

Configure context propagation

Restart your application and verify the data in Dynatrace

Configure data capture to meet privacy requirements

Instrument your application

C++ instrumentation requires some manual steps. Add the following to your CMakeLists.txt file and add the code snippet below to any C++ method you want to monitor.
Set names for the tracer, the span, and add attributes as you see fit.

cpp
find_package(CURL REQUIRED)
find_package(nlohmann_json REQUIRED)
find_package(opentelemetry-cpp CONFIG REQUIRED)
find_package(Protobuf REQUIRED)

target_include_directories(<YOUR_EXE_NAME> PRIVATE ${OPENTELEMETRY_CPP_INCLUDE_DIRS})

target_link_libraries(<YOUR_EXE_NAME> ${OPENTELEMETRY_CPP_LIBRARIES})

cpp
#include "opentelemetry/trace/provider.h"

namespace trace = opentelemetry::trace;

// your method
{
    auto provider = trace::Provider::GetTracerProvider();

    auto span = trace::Scope(provider->GetTracer("manual")
        ->StartSpan("my-span", { //TODO Replace with the name of your span
            {"my-key-1", "my-value-1"} //TODO Add attributes
        }));

    auto scope = get_tracer("manual")->WithActiveSpan(span);

    // your original code goes here

    span->End();
}

Send data to Dynatrace

With OneAgentWithout OneAgent

To send data to Dynatrace with OneAgent, you have to create and configure the following header file, and add it to your C++ application:

cpp
// tracer_common.h

#pragma once
#include "opentelemetry/exporters/otlp/otlp_http_exporter.h"
#include "opentelemetry/sdk/trace/samplers/always_on.h"
#include "opentelemetry/sdk/trace/simple_processor.h"
#include "opentelemetry/sdk/trace/tracer_provider.h"
#include "opentelemetry/trace/provider.h"

#include "opentelemetry/context/propagation/global_propagator.h"
#include "opentelemetry/context/propagation/text_map_propagator.h"
#include "opentelemetry/trace/propagation/http_trace_context.h"

#include "opentelemetry/ext/http/client/http_client.h"

#include <cstring>
#include <iostream>
#include <vector>
#include <fstream>
#include <list>

using namespace std;
namespace nostd    = opentelemetry::nostd;
namespace exporter = opentelemetry::exporter::otlp;
namespace sdktrace = opentelemetry::sdk::trace;
namespace resource = opentelemetry::sdk::resource;

namespace
{
  template <typename T>
  class HttpTextMapCarrier : public opentelemetry::context::propagation::TextMapCarrier
  {
  public:
    HttpTextMapCarrier<T>(T &headers) : headers_(headers) {}
    HttpTextMapCarrier() = default;
    virtual nostd::string_view Get(nostd::string_view key) const noexcept override
    {
      std::string key_to_compare = key.data();
      if (key == opentelemetry::trace::propagation::kTraceParent)
      {
        key_to_compare = "Traceparent";
      }
      else if (key == opentelemetry::trace::propagation::kTraceState)
      {
        key_to_compare = "Tracestate";
      }
      auto it = headers_.find(key_to_compare);
      if (it != headers_.end())
      {
        return it->second;
      }
      return "";
    }

    virtual void Set(nostd::string_view key, nostd::string_view value) noexcept override
    {
      headers_.insert(std::pair<std::string, std::string>(std::string(key), std::string(value)));
    }

    T headers_;
  };

  exporter::OtlpHttpExporterOptions opts;

  void initTracer()
  {
    opts.url = "http://localhost:14499/otlp/v1/traces";

    opts.content_type  = exporter::HttpRequestContentType::kBinary;

    resource::ResourceAttributes resource_attributes = {
        {"service.name", "cpp-quickstart"}, //TODO Replace with the name of your application
        {"service.version", "1.0.1"} //TODO Replace with the version of your application
    };

    resource::ResourceAttributes dt_resource_attributes;

    try
    {
        for (string name : {"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"})
        {
            string file_path;
            ifstream dt_file;
            dt_file.open(name);
            if (dt_file.is_open())
            {
                string dt_metadata;
                ifstream dt_properties;
                while (getline(dt_file, file_path))
                {
                    dt_properties.open(file_path);
                    if (dt_properties.is_open())
                    {
                        while (getline(dt_properties, dt_metadata))
                        {
                            dt_resource_attributes.SetAttribute(
                                dt_metadata.substr(0, dt_metadata.find("=")),
                                dt_metadata.substr(dt_metadata.find("=") + 1)
                            );
                        }
                        dt_properties.close();
                    }
                }
                dt_file.close();
            }
        }
    }
    catch (...) {}

    auto dt_resource = resource::Resource::Create(dt_resource_attributes);
    auto resource = resource::Resource::Create(resource_attributes);
    auto merged_resource = dt_resource.Merge(resource);

    auto exporter  = std::unique_ptr<sdktrace::SpanExporter>(
        new exporter::OtlpHttpExporter(opts));

    auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
        new sdktrace::SimpleSpanProcessor(std::move(exporter)));
    std::vector<std::unique_ptr<sdktrace::SpanProcessor>> processors;
    processors.push_back(std::move(processor));

    auto context = std::make_shared<sdktrace::TracerContext>(
      std::move(processors),
      merged_resource,
      std::unique_ptr<sdktrace::AlwaysOnSampler>(new sdktrace::AlwaysOnSampler));

    auto provider = nostd::shared_ptr<opentelemetry::trace::TracerProvider>(
          new sdktrace::TracerProvider(context));

    opentelemetry::trace::Provider::SetTracerProvider(provider);

    opentelemetry::context::propagation::GlobalTextMapPropagator::SetGlobalPropagator(
        nostd::shared_ptr<opentelemetry::context::propagation::TextMapPropagator>(
            new opentelemetry::trace::propagation::HttpTraceContext()));
  }

  nostd::shared_ptr<opentelemetry::trace::Tracer> get_tracer(std::string tracer_name)
  {
    auto provider = opentelemetry::trace::Provider::GetTracerProvider();
    return provider->GetTracer(tracer_name);
  }
}

When using OneAgent, make sure to enable the public Extension Execution Controller in your Dynatrace Settings, otherwise no data will be sent.

In the Dynatrace menu, go to Settings > Preferences > Extension Execution Controller. The toggles Enable Extension Execution Controller and Enable local PIPE/HTTP metric and Log Ingest API should be active.

To send data to Dynatrace without OneAgent, you have to create and configure the following header file, and add it to your C++ application:

cpp
// tracer_common.h

#pragma once
#include "opentelemetry/exporters/otlp/otlp_http_exporter.h"
#include "opentelemetry/sdk/trace/samplers/always_on.h"
#include "opentelemetry/sdk/trace/simple_processor.h"
#include "opentelemetry/sdk/trace/tracer_provider.h"
#include "opentelemetry/trace/provider.h"

#include "opentelemetry/context/propagation/global_propagator.h"
#include "opentelemetry/context/propagation/text_map_propagator.h"
#include "opentelemetry/trace/propagation/http_trace_context.h"

#include "opentelemetry/ext/http/client/http_client.h"

#include <cstring>
#include <iostream>
#include <vector>
#include <fstream>
#include <list>

using namespace std;
namespace nostd    = opentelemetry::nostd;
namespace exporter = opentelemetry::exporter::otlp;
namespace sdktrace = opentelemetry::sdk::trace;
namespace resource = opentelemetry::sdk::resource;

namespace
{
  template <typename T>
  class HttpTextMapCarrier : public opentelemetry::context::propagation::TextMapCarrier
  {
  public:
    HttpTextMapCarrier<T>(T &headers) : headers_(headers) {}
    HttpTextMapCarrier() = default;
    virtual nostd::string_view Get(nostd::string_view key) const noexcept override
    {
      std::string key_to_compare = key.data();
      if (key == opentelemetry::trace::propagation::kTraceParent)
      {
        key_to_compare = "Traceparent";
      }
      else if (key == opentelemetry::trace::propagation::kTraceState)
      {
        key_to_compare = "Tracestate";
      }
      auto it = headers_.find(key_to_compare);
      if (it != headers_.end())
      {
        return it->second;
      }
      return "";
    }

    virtual void Set(nostd::string_view key, nostd::string_view value) noexcept override
    {
      headers_.insert(std::pair<std::string, std::string>(std::string(key), std::string(value)));
    }

    T headers_;
  };

  exporter::OtlpHttpExporterOptions opts;

  void initTracer()
  {
    opts.url = "<URL>"; //TODO Replace <URL> to your SaaS/Managed-URL as mentioned in the next step

    opts.content_type  = exporter::HttpRequestContentType::kBinary;
    opts.http_headers.insert(
        std::make_pair<const std::string, std::string>("Authorization", "Api-Token <TOKEN>")); //TODO Replace <TOKEN> with your API Token as mentioned in the next step

    resource::ResourceAttributes resource_attributes = {
        {"service.name", "cpp-quickstart"}, //TODO Replace with the name of your application
        {"service.version", "1.0.1"} //TODO Replace with the version of your application
    };

    resource::ResourceAttributes dt_resource_attributes;

    try
    {
        for (string name : {"dt_metadata_e617c525669e072eebe3d0f08212e8f2.properties", "/var/lib/dynatrace/enrichment/dt_metadata.properties"})
        {
            string file_path;
            ifstream dt_file;
            dt_file.open(name);
            if (dt_file.is_open())
            {
                string dt_metadata;
                ifstream dt_properties;
                while (getline(dt_file, file_path))
                {
                    dt_properties.open(file_path);
                    if (dt_properties.is_open())
                    {
                        while (getline(dt_properties, dt_metadata))
                        {
                            dt_resource_attributes.SetAttribute(
                                dt_metadata.substr(0, dt_metadata.find("=")),
                                dt_metadata.substr(dt_metadata.find("=") + 1)
                            );
                        }
                        dt_properties.close();
                    }
                }
                dt_file.close();
            }
        }
    }
    catch (...) {}

    auto dt_resource = resource::Resource::Create(dt_resource_attributes);
    auto resource = resource::Resource::Create(resource_attributes);
    auto merged_resource = dt_resource.Merge(resource);

    auto exporter  = std::unique_ptr<sdktrace::SpanExporter>(
        new exporter::OtlpHttpExporter(opts));

    auto processor = std::unique_ptr<sdktrace::SpanProcessor>(
        new sdktrace::SimpleSpanProcessor(std::move(exporter)));
    std::vector<std::unique_ptr<sdktrace::SpanProcessor>> processors;
    processors.push_back(std::move(processor));

    auto context = std::make_shared<sdktrace::TracerContext>(
      std::move(processors),
      merged_resource,
      std::unique_ptr<sdktrace::AlwaysOnSampler>(new sdktrace::AlwaysOnSampler));

    auto provider = nostd::shared_ptr<opentelemetry::trace::TracerProvider>(
          new sdktrace::TracerProvider(context));

    opentelemetry::trace::Provider::SetTracerProvider(provider);

    opentelemetry::context::propagation::GlobalTextMapPropagator::SetGlobalPropagator(
        nostd::shared_ptr<opentelemetry::context::propagation::TextMapPropagator>(
            new opentelemetry::trace::propagation::HttpTraceContext()));
  }

  nostd::shared_ptr<opentelemetry::trace::Tracer> get_tracer(std::string tracer_name)
  {
    auto provider = opentelemetry::trace::Provider::GetTracerProvider();
    return provider->GetTracer(tracer_name);
  }
}

Lastly, you need to define the correct endpoint and token, to make sure your data arrives where it should be.

To set the endpoint:
1. Use your Environment ID to set the endpoint to which your app will send traces as follows:
  - Dynatrace SaaS https://{your-environment-id}.live.dynatrace.com/api/v2/otlp/v1/traces
  - Dynatrace Managed https://{your-domain}/e/{your-environment-id}/api/v2/otlp/v1/traces
  - Dynatrace ActiveGate https://{your-activegate-endpoint}/e/{your-environment-id}/api/v2/otlp/v1/traces
    - You may need to include the port to your ActiveGate endpoint. For example:
      - https://{your-activegate-endpoint}:9999/e/{your-environment-id}/api/v2/otlp/v1/traces
    - If you are running a containerized ActiveGate, you need to use the FQDN of it. For example:
      - https://{your-activegate-service-name}.dynatrace.svc.cluster.local/e/{your-environment-id}/api/v2/otlp/v1/traces
2. Replace <URL> in the code snippet above with your endpoint.
To create an authentication token
1. In the Dynatrace menu, go to Access tokens and select Generate new token.
2. Provide a Token name.
3. In the Search scopes box, search for Ingest OpenTelemetry traces and select the checkbox.
4. Select Generate token.
5. Select Copy to copy the token to your clipboard.
6. Save the token in a safe place; you can't display it again.
7. Replace <TOKEN> in the code snippet above with your token.

Configure context propagation optional

If your application receives a request or calls other applications, you need to configure context propagation to make sure the spans are linked together.

Whenever receiving an incoming request, you need to extract the parent context and create the new span as a child of it. Here is how:

cpp
#include "opentelemetry/trace/context.h"
#include "tracer_common.h" // header file created in the step above

using namespace opentelemetry::trace;
namespace context = opentelemetry::context;

// your method
{
    StartSpanOptions options;
    options.kind          = SpanKind::kServer;

    // extract context from http header
    HttpTextMapCarrier<http_client::Headers> carrier;
    auto prop        = context::propagation::GlobalTextMapPropagator::GetGlobalPropagator();
    auto current_ctx = context::RuntimeContext::GetCurrent();
    auto new_context = prop->Extract(carrier, current_ctx);
    options.parent   = GetSpan(new_context)->GetContext();

    // start scoped span with parent context extracted from http header
    auto span = get_tracer("manual-server")
                  ->StartSpan("my-server-span", { //TODO Replace with the name of your span
                              {"my-server-key-1", "my-server-value-1"} //TODO Add attributes
                            }, options);

    auto scope = get_tracer("http_server")->WithActiveSpan(span);

    // your code goes here

    span->End();
}

If your application calls another service, you need to ensure that you propagate the context, adding it to your outgoing request. Here is how:

cpp
#include "opentelemetry/ext/http/client/http_client_factory.h"
#include "opentelemetry/ext/http/common/url_parser.h"
#include "opentelemetry/trace/context.h"
#include "tracer_common.h" // header file created in the step above

using namespace opentelemetry::trace;
namespace context = opentelemetry::context;
namespace http_client = opentelemetry::ext::http::client;

// your method
{
    auto http_client = http_client::HttpClientFactory::CreateSync();
    std::string url  = "<HTTP_URL>";

    // start scoped active span
    StartSpanOptions options;
    options.kind = SpanKind::kClient;
    opentelemetry::ext::http::common::UrlParser url_parser(url);

    std::string span_name = url_parser.path_; // Fetch the URL path to be used as span name
    auto span = get_tracer("http-client")
                  ->StartSpan(span_name, {
                              {"my-client-key-1", "my-client-value-1"} //TODO Add attributes
                            }, options);

    auto scope = get_tracer("http-client")->WithActiveSpan(span);

    // inject current context into http header
    auto current_ctx = context::RuntimeContext::GetCurrent();
    HttpTextMapCarrier<http_client::Headers> carrier;
    auto prop = context::propagation::GlobalTextMapPropagator::GetGlobalPropagator();
    prop->Inject(carrier, current_ctx);

    // send http request
    http_client::Result result = http_client->Get(url, carrier.headers_);

    //your code goes here

    span->End();
}

Verify that the traces are ingested into Dynatrace

A few minutes after restarting your app, look for your spans:

In the Dynatrace menu, go to Distributed traces and select the Ingested traces tab.
- If you use OneAgent, go to Distributed traces and select the PurePaths tab.
Your spans will be part of an existing PurePath, if the root of your call is already monitored by OneAgent.

If your application does not receive any traffic, there will be no traces.

Configure data capture to meet privacy requirements

While Dynatrace automatically captures all OpenTelemetry resource and span attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data that's stored.

To view your custom span attributes, you need to allow them in the Dynatrace web UI first:

Span attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Span attributes.
Resource attributes In the Dynatrace menu, go to Settings and select Server-side service monitoring > Resource attributes.