Trace .NET Lambda functions

Dynatrace uses OpenTelemetry to monitor AWS Lambda invocations.

Prerequisites

Ensure that you have followed the initial configuration steps described in Configuration for monitoring AWS Lambda with OpenTelemetry before using the packages below.

The following NuGet packages can be used to cover different aspects of AWS Lambda tracing:

recommended OpenTelemetry.Instrumentation.AWSLambda version 1.1.0-beta.2+: Contains methods to trace incoming AWS Lambda invocations.
recommended OpenTelemetry.Contrib.Instrumentation.AWS version 1.0.1+: Traces outgoing AWS SDK calls to other AWS Lambda invocations and calls to AWS services like DynamoDB and SQS.
required Dynatrace.OpenTelemetry.Instrumentation.AwsLambda: Enables linking traces from one Lambda to another over an AWS SDK Lambda invocation. This is the only way to link these kinds of requests; however, if you don't need this linking, or don't use the AWS Lambda client SDK to invoke or receive other Lambda invocations, you can leave out the package.

Installation

Any of the above-listed packages can be installed via the CLI. For example, Dynatrace.OpenTelemetry.Instrumentation.AwsLambda can be installed using the following command:

bash
dotnet add package Dynatrace.OpenTelemetry.Instrumentation.AwsLambda

Note: Some packages may require you to specify a version explicitly or use the --prerelease command line flag, such as dotnet add package --prerelease OpenTelemetry.Instrumentation.AWSLambda.

Initialization

The initialization code for AWS Lambda tracing in your Function.cs file could look as follows (where Function is the configured Lambda handler class):

c#
using System.Threading.Tasks;
using Amazon.Lambda.Core;
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;
using OpenTelemetry;
using OpenTelemetry.Instrumentation.AWSLambda;
using OpenTelemetry.Trace;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Examples.AwsFunctionApp
{
    public class Function
    {
        private static readonly TracerProvider TracerProvider;

        static Function()
        {
            DynatraceSetup.InitializeLogging();
            TracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddDynatrace()
                // Configures AWS Lambda invocations tracing
                .AddAWSLambdaConfigurations(c => c.DisableAwsXRayContextExtraction = true)
                // Instrumentation for creation of span (Activity) representing AWS SDK call.
                // Can be omitted if there are no outgoing AWS SDK calls to other AWS Lambdas and/or calls to AWS services like DynamoDB and SQS.
                .AddAWSInstrumentation(c => c.SuppressDownstreamInstrumentation = true)
                // Adds injection of Dynatrace-specific context information in certain SDK calls (e.g. Lambda Invoke).
                // Can be omitted if there are no outgoing calls to other Lambdas via the AWS Lambda SDK.
                .AddDynatraceAwsSdkInjection()
                .Build();
        }
    }
}

Setting the option DisableAwsXRayContextExtraction to true is required to skip Amazon X-Ray parent extraction, which may conflict with the Dynatrace propagation.

Tracing incoming AWS Lambda calls

Example 1: Trace an AWS Lambda invoked via AWS SDK

In addition to the initialization part provided above, the handler method of a Lambda invoked via AWS SDK could look as follows:

c#
using System.Threading.Tasks;
using Amazon.Lambda.Core;
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;
using OpenTelemetry;
using OpenTelemetry.Instrumentation.AWSLambda;
using OpenTelemetry.Trace;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Examples.AwsFunctionApp
{
    public class Function
    {
        private static readonly TracerProvider TracerProvider;

        // Use initialization code from the "Initialization" section of the docs

        public Task FunctionHandlerAsync(object input, ILambdaContext context)
        {
            var propagationContext = AwsLambdaHelpers.ExtractPropagationContext(context);
            return AWSLambdaWrapper.TraceAsync(TracerProvider, FunctionHandlerInternalAsync, input, context, propagationContext.ActivityContext);
        }

        private Task FunctionHandlerInternalAsync(object input, ILambdaContext context)
        {
            // This is just an example of function handler and should be replaced by actual code.

            return Task.CompletedTask;
        }
    }
}

A parent context is extracted explicitly using the AwsLambdaHelpers class from the Dynatrace.OpenTelemetry.Instrumentation.AwsLambda package.
An activity tracing the incoming request and the handler is created by the TraceAsync method.
TraceAsync should be used when you trace an async function or a function returning a task. That way, the activity ends only when the task completes.

Example 2: Trace an AWS Lambda invoked via Amazon API Gateway (incoming HTTP request)

In addition to the initialization part provided above, the Lambda handler invoked via Amazon API Gateway could look as follows:

c#
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;
using OpenTelemetry;
using OpenTelemetry.Instrumentation.AWSLambda;
using OpenTelemetry.Trace;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Examples.AwsFunctionApp
{
    public class Function
    {
        private static readonly TracerProvider TracerProvider;

        // Use initialization code from the "Initialization" section of the docs

        public APIGatewayHttpApiV2ProxyResponse FunctionHandler(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
        {
            return AWSLambdaWrapper.Trace(TracerProvider, FunctionHandlerInternal, request, context);
        }

        private APIGatewayHttpApiV2ProxyResponse FunctionHandlerInternal(APIGatewayHttpApiV2ProxyRequest request, ILambdaContext context)
        {
            // This is just an example of function handler and should be replaced by actual code.

            return new APIGatewayHttpApiV2ProxyResponse
            {
                StatusCode = 200,
                Body = "Example function result",
            };
        }
    }
}

A parent context is extracted from the incoming request in the Trace (or TraceAsync) method.
An activity tracing the incoming request and the handler is created by the Trace method.
In general, the Trace/TraceAsync methods support any trigger, but extended support is available for the APIGatewayProxyRequest and APIGatewayHttpApiV2ProxyRequest trigger types. For more details about request/response types, consult the GitHub documentation.
Trace should only be used when you have a function returning something other than a Task. For the asynchronous handler, TraceAsync should be used instead.

Example 3: Tracing without the `AwsLambda` package

If you prefer not to use the OpenTelemetry.Instrumentation.AWSLambda package, you can manually create an activity for Lambda. Note that this involves quite a bit of work, as Dynatrace requires certain activity tags (span attributes) to detect the service (conforming to the OpenTelemetry FaaS trace conventions and resource conventions). You also need to manually extract the parent context.

For this example, only the Dynatrace.OpenTelemetry.Instrumentation.AwsLambda package is required.

c#
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Reflection;
using Amazon.Lambda.APIGatewayEvents;
using Amazon.Lambda.Core;
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AwsLambda;
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;
using OpenTelemetry.Trace;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace Examples.AwsFunctionApp
{
    public class Function
    {
        private static readonly TracerProvider TracerProvider;
        private static readonly ActivitySource ActivitySource;

        static Function()
        {
            DynatraceSetup.InitializeLogging();
            var activitySourceName = Assembly.GetExecutingAssembly().GetName().Name;
            ActivitySource = new ActivitySource(activitySourceName);
            TracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddSource(activitySourceName)
                .AddDynatrace()
                .Build();
        }

        public static IEnumerable<KeyValuePair<string, object>> GetFunctionTags(ILambdaContext context, string trigger)
        {
            return new KeyValuePair<string, object>[]
            {
                new("faas.name", context.FunctionName),
                new("faas.id", context.InvokedFunctionArn),
                new("faas.trigger", trigger),
                new("cloud.platform", "aws_lambda"),
                new("cloud.provider", "aws"),
                new("cloud.region", Environment.GetEnvironmentVariable("AWS_REGION")),
            };
        }

        public APIGatewayProxyResponse FunctionHandler(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, ILambdaContext context)
        {
            try
            {
                var parentContext = ExtractParentContext(apiGatewayProxyEvent, context);
                using var activity = ActivitySource.StartActivity(ActivityKind.Server, parentContext, GetFunctionTags(context, "http"));
                return new APIGatewayProxyResponse
                {
                    StatusCode = 200,
                    Body = "Example function result",
                };
            }
            catch (Exception ex)
            {
                context.Logger.LogLine($"Exception occurred while handling request: {ex.Message}");
                throw;
            }
            finally
            {
                TracerProvider?.ForceFlush();
            }
        }

        private static ActivityContext ExtractParentContext(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, ILambdaContext context)
        {
            var propagationContext = AwsLambdaHelpers.ExtractPropagationContext(context);
            if (propagationContext == default)
            {
                propagationContext = Propagators.DefaultTextMapPropagator.Extract(default, apiGatewayProxyEvent, HeaderValuesGetter);
            }

            return propagationContext.ActivityContext;
        }

        private static IEnumerable<string> HeaderValuesGetter(APIGatewayHttpApiV2ProxyRequest apiGatewayProxyEvent, string name) =>
            (apiGatewayProxyEvent.Headers != null && apiGatewayProxyEvent.Headers.TryGetValue(name.ToLowerInvariant(), out var value)) ? new[] { value } : null;
    }
}

Amazon DynamoDB operations tracing

To trace DynamoDB operations, add the OpenTelemetry.Contrib.Instrumentation.AWS dependency—the AWS client instrumentation for OpenTelemetry .NET—to your project:

bash
dotnet add package OpenTelemetry.Contrib.Instrumentation.AWS

After you add the dependency, update the initialization code by adding a call to the AddAWSInstrumentation() extension method:

c#
using Dynatrace.OpenTelemetry;
using OpenTelemetry;
using OpenTelemetry.Trace;
using System.Reflection;

namespace Examples.AwsFunctionApp
{
    public class Function
    {
        private static readonly TracerProvider TracerProvider;

        static Function()
        {
            DynatraceSetup.InitializeLogging();
            var activitySourceName = Assembly.GetExecutingAssembly().GetName().Name;
            TracerProvider = Sdk.CreateTracerProviderBuilder()
                .AddSource(activitySourceName)
                // Instrumentation used for tracing calls to AWS services via AWS SDK (including DynamoDB)
                .AddAWSInstrumentation(c => c.SuppressDownstreamInstrumentation = true)
                .Build();
        }
    }
}

Note: If you prefer not to see HTTP child nodes under AWS SDK calls, set the SuppressDownstreamInstrumentation option to true.

In the following example, the DynamoDB operations DescribeTable, UpdateItem, and GetItem are represented as separate span child nodes with a common parent outbound-aws-dotnet in eu-central-1 (Lambda function performing DynamoDB operations):