Use OpenTelemetry to trace AWS Lambda .NET Core functions

In Feb 2021, AWS announced Support for AWS Distro for OpenTelemetry .NET Tracing. For general information about AWS Distro for OpenTelemetry, see the AWS Distro for OpenTelemetry page.

For tracing AWS Lambda for other languages such as Java, Node.JS, and Python using the Dynatrace AWS Lambda Layer, see Deploy OneAgent as Lambda extension.

Setup

The OpenTelemetry OTLP exporter for .NET currently only supports the GRPC-based format. To ingest the GRPC format using the Dynatrace Trace API, you must use an OpenTelemetry collector between the exporter and Dynatrace.

You can either choose to self-host a collector or use the AWS Distro for OpenTelemetry Collector (ADOT Collector).

Add the ARN of the Lambda Layer ADOT Collector

AWS Region

Lambda layers are a rationalized resource, meaning that they can only be used in the Region in which they are published. Make sure to use the layer in the same region as your Lambda functions.

Collector layer: aws-otel-collector-ver-0-27-0.

For the complete list of the AWS-managed OpenTelemetry Lambda layers see AWS Distro for OpenTelemetry - AWS Lambda respository

Lambda layer ARN format is:

arn:aws:lambda:\<region>:901920570463:layer:<layer>:1

Configure ADOT Collector

The configuration of the ADOT Collector follows the OpenTelemetry standard.

By default, the ADOT Lambda layer uses the config.yaml, which exports OpenTelemetry data to AWS X-Ray. To export the data to Dynatrace, you need to customize the configuration using the OpenTelemetry OTLP exporter.

To customize the collector configuration, add a configuration YAML file to your function code. Once the file has been deployed with a Lambda function, create an environment variable on your Lambda function OPENTELEMETRY_COLLECTOR_CONFIG_FILE and set it to /var/task/*<path/<to>/<filename>*. This will tell the extension where to find the collector configuration.

Here is a sample configuration file, collector.yaml in the root directory:

  1. Copy collector.yaml in the root directory
  2. Set an environemnt variable OPENTELEMETRY_COLLECTOR_CONFIG_FILE to /var/task/<path/<to>/<file>
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  otlphttp:
    endpoint: "https://<YOUR-TENANT-ID>.live.dynatrace.com/api/v2/otlp"
    headers: {"Authorization": "Api-Token <YOUR-DYNATRACE-API-TOKEN>"}
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

For further details on configuration, see Ingest OpenTelemetry trace data.

You can set this via the Lambda console, or via the AWS CLI. With the CLI, use he following command:

aws lambda update-function-configuration --function-name Function --environment Variables={OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/collector.yaml}

You can configure environment variables via CloudFormation template as well:

Function:
  Type: AWS::Serverless::Function
  Properties:
    ...
    Environment:
      Variables:
        OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml

Code Instrumentation with OpenTelemetry

Add dependencies

Add following dependencies via NuGet to your project:

OpenTelemetry.Exporter.OpenTelemetryProtocol

If you are using the AWS SDK to interact with other AWS services, you can add auto-instrumentation using the ADOT SDK for .NET

OpenTelemetry.Contrib.Instrumentation.AWS

OpenTelemetry also provides other auto-instrumentation libraries available as NuGet packages

Add OpenTelemetry Tracer

AWS Distro currently doesn't provide a wrapper layer for .NET as it does for other languages. You have to add a tracer to your code and create a so-called root span.

The following sample uses an AWS Lambda Proxy Integration

public class Functions
{
  public Functions() {}

  //Defines the OpenTelemetry resource attribute "service.name" which is mandatory
  private const string servicename = "AWS Lambda";

  //Defines the OpenTelemetry Instrumentation Library.
  private const string activitySource = "MyCompany.MyProduct.MyLibrary";

  //Provides the API for starting/stopping activities.
  private static readonly ActivitySource MyActivitySource = new ActivitySource(activitySource);

  public async Task<APIGatewayProxyResponse> Get(APIGatewayProxyRequest request, ILambdaContext context)
  {
      AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport",true);

      //Initialize OpenTelemetry Tracer
      using (Sdk.CreateTracerProviderBuilder()
          .SetSampler(new AlwaysOnSampler())
          .AddSource(activitySource)
          .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(servicename))
          .AddAWSInstrumentation() //Add auto-instrumentation for AWS SDK
          .AddHttpClientInstrumentation() //Add auto-instrumentation for AWS SDK
          .AddOtlpExporter(otlpOptions =>
          {
              //use a local the local endpoint for AWS Lambda ADOT Collector Layer
              //or an endpoint configured via environment variable.
              var collectorUrl = Environment.GetEnvironmentVariable("COLLECTOR_URL") ?? "http://localhost:55680";
              otlpOptions.Endpoint = new Uri(collectorUrl);

          })
          .Build())
      {
          //create root-span, connecting with trace-parent read from the http-header
          using (var activity = MyActivitySource.StartActivity("Invoke", ActivityKind.Server, request.Headers["traceparent"]))
          {

            //.....
            //... YOUR CODE GOES HERE
            //....
          }
      }
  }
}