Trace Google Cloud Functions with OpenTelemetry Node.js

The @dynatrace/opentelemetry-gcf module provides APIs for tracing Node.js on Google Cloud Functions (GCF).

Prerequisites

Make sure you have followed the instructions on how to integrate OpenTelemetry on Google Cloud Functions.
So far, only HTTP triggers are supported.

Installation

To set up OpenTelemetry Node.js integration on Google Cloud Functions, run the command below in the root directory of your Google Cloud Function project.

bash
npm install --save @dynatrace/opentelemetry-gcf

This will install the latest version of the @dynatrace/opentelemetry-gcf module from NPM. Note that this library by itself is not enough to start tracing your Google Cloud Functions. See the Usage section below for the remaining required steps.

Usage

To export traces to Dynatrace

Select one of the two ways below to initialize tracing.

NodeTracerProvider used to initialize tracing is more lightweight than NodeSDK.
NodeSDK is typically used if you're interested in additional OpenTelemetry signals such as metrics.

Using NodeTracerProvider (recommended)Using NodeSDK

Install the required OpenTelemetry packages with the command below.

bash
npm install --save @opentelemetry/sdk-trace-node @opentelemetry/semantic-conventions

After you install the packages, initialize tracing using the following snippet as an example.

javascript
const { Resource } = require('@opentelemetry/resources');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { DtSpanExporter, DtSpanProcessor, DtTextMapPropagator, DtSampler } = require('@dynatrace/opentelemetry-core');

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "My Service",
  }),
  sampler: new DtSampler(),
  // ...other configurations
});

const processor = new DtSpanProcessor(new DtSpanExporter());
provider.addSpanProcessor(processor);
provider.register({
  propagator: new DtTextMapPropagator(),
  // ...other configurations
});

Install the required OpenTelemetry packages with the command below.

bash
npm install --save @opentelemetry/sdk-node @opentelemetry/semantic-conventions

After you install the packages, initialize tracing using the following snippet as an example.

javascript
const { Resource } = require('@opentelemetry/resources');
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { DtSpanExporter, DtSpanProcessor, DtTextMapPropagator, DtSampler } = require('@dynatrace/opentelemetry-core');

const sdk = new NodeSDK({
  resource: new Resource({
   [SemanticResourceAttributes.SERVICE_NAME]: "My Service"
  }),
  sampler: new DtSampler(),
  spanProcessor: new DtSpanProcessor(new DtSpanExporter()),
  textMapPropagator: new DtTextMapPropagator(),
  // ...other configurations
});

sdk.start().then(() => {
  // Resources have been detected and SDK is started
});

Start the root Google Cloud Function server span, using one of the two general patterns in OpenTelemetry below.

Start an active span (recommended)Start the span and activate it later

Example that starts and immediately activates a span inside a Google Cloud Function:

javascript
const { startActiveHttpSpan, endHttpSpanAndFlush } = require('@dynatrace/opentelemetry-gcf');

// ...tracing initialization code

async function handler(req, res) {
  await startActiveHttpSpan(req, async (span) => {
    let error;
    try {
      // do something
    } catch (e) {
      error = e;
    }

    // status should be set before span ends
    res.status(error != null ? 500 : 200);
    /**
     * Span must be ended and flushed before handler sends response.
     * This limitiation comes from GCF, for details see:
     * https://cloud.google.com/functions/docs/concepts/nodejs-runtime#signal-termination
     */
    await endHttpSpanAndFlush(span, res, error);
    res.send("hello world");
  });
}

Example that starts a span inside a Google Cloud Function and later activates it within the same function.

javascript
const { context, trace, ROOT_CONTEXT } = require('@opentelemetry/api');
const { startHttpSpan, endHttpSpanAndFlush } = require('@dynatrace/opentelemetry-gcf');

// ...tracing initialization code

async function handler(req, res) {
  const span = await startHttpSpan(req);

  let error;
  await context.with(trace.setSpan(ROOT_CONTEXT, span), async () => {
    try {
      // do something
    } catch (e) {
      error = e;
    }
  });

  // status should be set before span ends
  res.status(error != null ? 500 : 200);
  /**
   * Span must be ended and flushed before handler sends response.
   * This limitiation comes from GCF, for details see:
   * https://cloud.google.com/functions/docs/concepts/nodejs-runtime#signal-termination
   */
  await endHttpSpanAndFlush(span, res, error);
  res.send("hello world");
}

Cold start

Starting a Google Cloud Function span during cold starts produces additional HTTP requests to fetch metadata from your Google Cloud Platform environment and set the attributes required for Dynatrace to process the spans.

Span flush

You need to flush the spans before a function's response is sent to the client. For details on this limitation, see Signalling function termination.

You can use endHttpSpan() and flushSpans() separately instead of endHttpSpanAndFlush() when needed.

Note: Flushing spans in the function's code results in longer execution time, as this operation becomes part of the function's execution logic. Experiments have shown that, usually, not flushing spans explicitly results in correctly exported spans as well (enabled via continuous background export). However, since a Google Cloud Function could be suspended immediately after sending its response, it might result in span loss.

Caveats

You need to pay special attention to cases like unhandled exceptions or function timeouts. If not handled properly, they could lead to a non-ended, and therefore non-exported, span.

Dynatrace overhead

Because span export and metadata fetch take some time during cold starts, they increase the duration of the function and consequently increase costs.
Pay attention to infrequently invoked functions (usually with cold starts), which might require more time for the TCP handshake during the span export.
Any network problem between the exporter and Dynatrace backend might also lead to unexpected high overhead.