Instrument your Erlang application with OpenTelemetry

This walkthrough shows how to add observability to your Erlang application using the OpenTelemetry Erlang libraries and tools.

Prerequisites

Get the Dynatrace access details

Determine the API base URL

For details on how to assemble the base OTLP endpoint URL, see Export with OTLP.

The URL should end in /api/v2/otlp.

Get API access token

The access token for ingesting traces, logs, and metrics can be generated under Access Tokens.

Export with OTLP has more details on the format and the necessary access scopes.

Set up OpenTelemetry

Instrument your application

Add tracing

Spans are started with the macro with_span and accept an optional list of span attributes, as well as the code block for this span. The span will automatically finish when the code block returns.

-export([init/2]).

-include_lib("opentelemetry_api/include/otel_tracer.hrl").
-include_lib("opentelemetry/include/otel_resource.hrl").

init( Req, State ) ->
    ?with_span(<<"parent_span">>, #{attributes => [ %%TODO Add span name
      {<<"my-key-1">>, <<"my-value-1">>}] %%TODO Add attributes at span creation
    }, fun child_function/1),
    %% Your code goes here


child_function(_SpanCtx) ->
    ?with_span(<<"child_span">>, #{},
               fun(_ChildSpanCtx) ->
                ?set_attributes([{<<"child-key-1">>, <<"child-value-1">>}]) %%TODO Add attributes after span creation
               end).

Collect metrics

No example yet, as OpenTelemetry for Erlang does not have stable support for metrics yet.

Connect logs

No example yet, as OpenTelemetry for Erlang does not have stable support for logs yet.

Depending on the status of the OpenTelemetry SDK, the pre-release version may nonetheless already allow the ingestion of your logs.

Ensure context propagation optional

Context propagation is particularly important when network calls (for example, REST) are involved.

Extracting the context when receiving a request

For extracting information on an existing context, we pass the headers to the otel_propagator_text_map.extract function, which parses the context information provided by the headers and sets the current context based on that.

%% Get Headers from incoming request
Headers = maps:get(headers, Req),
otel_propagator_text_map:extract(maps:to_list(Headers)),

SpanCtx = ?start_span(<<"span-name">>),

%% As we used `otel_propagator_text_map` the current context is from the parent span
Ctx = otel_ctx:get_current(),

proc_lib:spawn_link(fun() ->
    %% Start span and set as current
    otel_ctx:attach(Ctx),
    ?set_current_span(SpanCtx),

    %% Create response
    Resp = cowboy_req:reply(
        200,
        #{<<"content-type">> => <<"application/json">>},
        <<"{\"message\": \"hello world\"}">>,
        Req
    ),

    {ok, Resp, State},
    ?end_span(SpanCtx)

Injecting the context when sending requests

The following example uses otel_propagator_text_map:inject to provide the HTTP headers (necessary for context propagation) in NewHeaders, which we eventually merge with the existing header object Headers and pass to the httpc:request call, which allows the receiving endpoint to continue the trace with the provided information.

?with_span(<<"span-name">>, #{},
    fun(_ChildSpanCtx) ->

        %% A custom header example
        Headers = [{"content-type", "application/json"}, {"X-Custom-Header", "some-value"}],

        %% We convert the traceparent information and merge the 2 headers as
        %% Httpc:request requires tuples of strings
        Tmp = [],
        NewHeaders = headers_list(otel_propagator_text_map:inject(opentelemetry:get_text_map_injector(), Tmp)),
        MergedHeaders = lists:append(Headers, NewHeaders),

        {ok, Res} = httpc:request(get, {URL, MergedHeaders}, [], []),
        io:format("Response: ~p~n", [Res])
    end).

headers_list(Headers) ->
    [{binary_to_list(Name), binary_to_list(Value)} || {Name, Value} <- Headers].

Configure data capture to meet privacy requirements optional

While Dynatrace automatically captures all OpenTelemetry attributes, only attribute values specified in the allowlist are stored and displayed in the Dynatrace web UI. This prevents accidental storage of personal data, so you can meet your privacy requirements and control the amount of monitoring data stored.

To view your custom attributes, you need to allow them in the Dynatrace web UI first. To learn how to configure attribute storage and masking, see Attribute redaction.

Verify data ingestion into Dynatrace

Once you have finished the instrumentation of your application, perform a couple of test actions to create and send demo traces, metrics, and logs and verify that they were correctly ingested into Dynatrace.

To do that for traces, go to Distributed Traces and select the Ingested traces tab. If you use OneAgent, select PurePaths instead.

For metrics and logs, go to Metrics or Logs or Logs & Events (latest Dynatrace).