Trace Azure Functions written in .NET

Prerequisites

Ensure that you have followed the initial configuration steps described in Set up OpenTelemetry monitoring for Azure Functions on Consumption Plan before using the packages below.

Installation

Add the following dependencies to your project:
- required Dynatrace.OpenTelemetry - Provides integration of Dynatrace-specific components (for activity export and propagation) into OpenTelemetry .NET. The minimum (referenced) OpenTelemetry version is currently 1.1.0.
- optional OpenTelemetry.Extensions.Hosting - Uses a TracerProvider with a dependency injection. Currently only available as a pre-release (release candidate).
Example commands to add dependencies:
```
bash
dotnet add package Dynatrace.OpenTelemetry
dotnet add package --prerelease OpenTelemetry.Extensions.Hosting
```
Additionally, depending on the runtime you use, we recommend that you use the following Azure-functions helper packages:
- recommended For in-process (library) functions (dotnet runtime):
  - Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions
  - OpenTelemetry.Instrumentation.AspNetCore
  To add the packages, run the command below.
```
bash
dotnet add package Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions
dotnet add package --prerelease OpenTelemetry.Instrumentation.AspNetCore
```
- recommended For isolated aka worker functions (dotnet-isolated runtime):
  - Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions.Worker
  To add the package, run the command below.
```
bash
dotnet add package Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions.Worker
```
- optional Alternatively, on both runtimes you can use a Dynatrace package called Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions.Core with the following functions:
  - AzureFunctionsCoreInstrumentation.Trace
  - AzureFunctionsCoreInstrumentation.TraceAsync

Example using the dotnet (in-process) runtime

Your Startup.cs could look as follows:

c#
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions;
using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection;
using OpenTelemetry.Trace;

[assembly: FunctionsStartup(typeof(Examples.AzureFunctionApp.Startup))]

namespace Examples.AzureFunctionApp
{
    internal class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            builder.Services.AddOpenTelemetryTracing(sdk => sdk
                .AddAzureFunctionsInstrumentation()
                .AddAspNetCoreInstrumentation()
                // ... any custom OTel setup ...
                .AddDynatrace()
                // ... if you need custom resources, set them after AddDynatrace & call AddTelemetrySdk (see below)
              );
        }
    }
}

An instrumented in-process function could look like this:

c#
using System.Diagnostics;
using System.Threading.Tasks;
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;

namespace Examples.AzureFunctionApp
{
    public class Function
    {
        public Function(ILoggerFactory loggerFactory)
        {
            // This is needed in every function in your app.
            DynatraceSetup.InitializeLogging(loggerFactory);
        }

        [FunctionName("MyFunction")]
        public async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest request,
            Microsoft.Azure.WebJobs.ExecutionContext ctx)
        {
            // This adds the required attributes to make the activity recognizable as an Azure function invocation.
            // Put this line first - there should be minimal time elapsing between the Activity being created
            // by the ASP.NET core instrumentation and the call to this method.
            AzureFunctionsInstrumentation.AddIncomingHttpAzureFunctionCallInfo(Activity.Current, ctx);

            // Your handler code...
        }
    }
}

Additionally, you need to modify host.json to allow logging for Dynatrace.OpenTelemetry. Note that this does not enable logging unless explicitly configured. See InitializeLogging.

json
{
  "version": "2.0",
  "logging": {
    // ...
   "logLevel": {
      "Dynatrace.OpenTelemetry": "Debug"
    }
  }
}

Example using the dotnet-isolated runtime

Your Program.cs could look as follows:

c#
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions;
using Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions.Worker;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

DynatraceSetup.InitializeLogging();

var host = new HostBuilder()
    .ConfigureFunctionsWorkerDefaults(fw => fw.UseTracingMiddleware())
    .ConfigureServices(services => services
        .AddOpenTelemetryTracing(tracing => tracing
            .AddAzureFunctionsInstrumentation()
            // ... any custom OTel setup ...
            .AddDynatrace()
            // ... if you need custom resources, set them after AddDynatrace (see below)
        ))
    .Build();

host.Run();

No additional code is needed to instrument functions; everything is handled by the middleware.

Technical details

`InitializeLogging`

Calling InitializeLogging is required even if you don't plan to enable logging, and the actual log messages won't be logged even after calling this method, unless configured.
If you use the dotnet-isolated runtime (out-of-process, worker functions), you need to call InitializeLogging in your Main method before calling AddDynatrace. You can pass null as loggerFactory, so that, if enabled, logging can use Console.Out/Console.Error. This is automatically forwarded to AppInsights for the dotnet-isolated runtime.
If you have specific requirements, you can also pass any custom LoggingFactory.
For the dotnet runtime (in-process, class-library), sending logs to AppInsights requires using an ILogger or ILoggerFactory injected into the function with a dependency injection. Thus, you shouldn't use null as an argument for the loggerFactory parameter, but call InitializeLogging the first time any function in your Function App is invoked. To get the ILoggerFactory, simply add a parameter of that type.
If you use the ILoggerFactory provided by Azure Functions, you also need to modify host.json to enable logging there. We recommend that you always use the debug log-level in host.json, as the actual log messages handed to the ILogger are separately configured in the Dynatrace configuration.
```
json
{
  "version": "2.0",
  "logging": {
    // ...
    "logLevel": {
      "Dynatrace.OpenTelemetry": "Debug"
    }
  }
}
```

`AddDynatrace`

AddDynatrace is an extension method to OpenTelemetry's TracerProvider. It requires using Dynatrace.OpenTelemetry. Currently, there aren't any additional parameters for this function, as configuration is read from environment variables and a dtconfig.json file. For details, see Set up OpenTelemetry monitoring for Azure Functions on Consumption Plan.
AddDynatrace mainly adds an ActivityProcessor to the TracerProvider that will send all activities to Dynatrace. This extension:
- Sets the resources required by Dynatrace. Due to an issue with the OpenTelemetry .NET SDK, this will override any existing resources. If you need custom resources, you need to call SetResourceBuilder on the TracerProvider after AddDynatrace. Be aware that this will override the resources configured by AddDynatrace and you need to readd them as part of the same SetResourceBuilder call. You can do this by calling the OpenTelemetry SDK's AddTelemetrySdk extension method on the ResourceBuilder.
- Exchanges the global Propagators.DefaultTextMapPropagator with a custom one that is based on the default W3C-format, but does additional processing of tracestate and additional Dynatrace-specific HTTP headers. The baggage propagator is also enabled, as is default for OpenTelemetry .NET. There's currently no way to disable it. Using another propagator isn't supported and will lead to missing links in the distributed traces.

The following minimal snippet might be used to initialize a TracerProvider with AddDynatrace:

c#
using Dynatrace.OpenTelemetry;
using Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions;
using OpenTelemetry.Trace;

// ...
// (call DynatraceSetup.InitializeLogging before or after AddDynatrace depending on runtime)
// ...

TracerProvider tracerProvider = Sdk.CreateTracerProviderBuilder().AddDynatrace().Build();

`AzureFunctionsCoreInstrumentation.Trace`/`TraceAsync`

This is the low-level instrumentation function provided in the Dynatrace.OpenTelemetry.Instrumentation.AzureFunctions.Core package. You usually shouldn't use these, but instead use the runtime-specific helpers, as in the examples above.

This function creates and starts a System.Diagnostics.Activity and runs the handler function argument, then stops Activity and records any exception on it.

The parent ActivityContext must be extracted from the HTTP headers using the Propagators.DefaultTextMapPropagator, which AddDynatrace initializes. If you don't pass any parent, a root span will be created (Activity.Current won't be used).

This is how a parent could be manually extracted with in-process functions for use with AzureFunctionsCoreInstrumentation.Trace/TraceAsync when not using the ASP.NET core instrumentation:

c#
private static ActivityContext ExtractParentContext(HttpRequest request)
{
    var context = Propagators.DefaultTextMapPropagator.Extract(default, request, HeaderValuesGetter);
    return context.ActivityContext;
}

private static IEnumerable<string> HeaderValuesGetter(HttpRequest request, string name) =>
    request.Headers.TryGetValue(name, out var values) ? values : (IEnumerable<string>)null;

For worker functions, the code can become more complex because in addition to HTTP headers, you may want to (but don't have to) use the W3C TraceContext provided in the FunctionContext:

c#
private static ActivityContext ExtractParentContext(HttpRequestData request, FunctionContext context) {
    ActivityContext parent = default;

    PropagationContext ctx = Propagators.DefaultTextMapPropagator.Extract(
        default,
        request.Headers,
        (c, k) => c.TryGetValues(k, out var value) ? value : null);
    parent = ctx.ActivityContext;
    if (parent == default)
    {
        PropagationContext ctx2 = Propagators.DefaultTextMapPropagator.Extract(
            default,
            context.TraceContext,
            (c, k) =>
            {
                string? result =
                    k.Equals("traceparent", StringComparison.OrdinalIgnoreCase) ? c.TraceParent :
                    k.Equals("tracestate", StringComparison.OrdinalIgnoreCase) ? c.TraceState :
                    null;
                return result == null ? null : new[] { result };
            });
        parent = ctx2.ActivityContext;
    }
    return parent;
}

Instrumenting `HttpClient` calls (outgoing HTTP requests)

A very common need is to trace outgoing HTTP requests. This can be achieved by using the OpenTelemetry.Instrumentation.Http NuGet package (currently only available as a pre-release).

The instrumentation then has to be added to your TracerProvider setup by calling AddHttpClientInstrumentation, for example, in Program.cs:

c#
// ...
using OpenTelemetry.Trace;
// ...
var host = new HostBuilder()
    // ...
    .ConfigureServices(services => services
        .AddOpenTelemetryTracing(tracing => tracing
            // ...
            .AddHttpClientInstrumentation(op =>
            {
                // Exclude outgoing calls with no parent activity.
                op.Filter = req => Activity.Current?.Parent != null;
            })))
        .Build();
// ...

Using a request filter as in the example above is highly recommended, as otherwise, depending on your Function Apps configuration, you might observe a large number of periodic requests to https://rt.services.visualstudio.com/QuickPulseService.svc/ping or similar URLs.

Note: Because of an Azure Functions runtime issue, the HTTP instrumentation won't work on Azure Functions in-process version 3.

The underlying issue can also affect other instrumentations. Therefore, we do not recommend using in-process version 3 functions.