OpenTelemetry Collector

A collector is a proxy that can receive, process, and export telemetry data.

• A collector can receive data in various formats (such as Jaeger, Prometheus, and OTLP) and send it to one or more backends, including Dynatrace.
• A collector is also able to change the format of the received data to enable transfer between various services.
• Each collector component (receiver, processor, and exporter) is enabled through pipelines and can be defined via YAML.
• If the application is instrumented through the standardized OTLP format, no collector is necessary. However, you might want to use a Collector for scalability reasons.
  • For simply trying out and getting started with OpenTelemetry, sending your data directly to a backend is a great way to get value quickly. Also, in a development or small-scale environment you can get decent results without a collector.
  • In general using a Collector alongside your service can be an advantage, since it allows your service to offload data quickly and the collector can take care of additional handling like retries, batching, encryption or even sensitive data filtering.

Setup

Deployment

Configuration

Deployment

A collector can be deployed as an agent or a gateway.

• As an agent, the collector is deployed either with the application or on the same host as the application (recommended). This collector can receive telemetry data and enhance it with, for example, tags or infrastructure information.
• As a gateway, one or more collector instances can be deployed as standalone services. This collector can be deployed additionally, for example, per cluster, region, or data center. A load balancer can help scale the independently operating collector instances.

Depending on your system, you can set up a collector in one of six ways:

1. Docker (from Docker Hub or through a local example)
2. Kubernetes
3. Nomad
4. Linux Packaging
5. Windows Packaging
6. Local

The OpenTelemetry project provides two versions of the collector:

• Core: Foundational components such as configuration and applicable receivers, processors, exporters, and extensions; offers support for popular open-source projects including Jaeger, Prometheus, and Fluent Bit.
• Contrib: All the components of the core version plus optional or experimental components; also contains more specialized or vendor-specific receivers, processors, exporters, and extensions.

If you are looking to use additional features or exporters (e.g. the Dynatrace Exporter), you need to import those from the Contrib repository. Double check all the imports on the application files in which you are using such features.

Configuration

To successfully configure your collector, you need to configure each component (receiver, processor, and exporter) individually in a YAML file.

Receiver

A receiver is a component that enables data to come into the collector. It can receive data from multiple data sources. Many receivers come with default settings and do not need much configuration besides the specified name.

For a list of available receivers and their basic configuration, see the official OpenTelemetry documentation.

Processor optional

A processor is a component that decides what to do with the data. Though there are some recommended processors, they are optional.

For a list of available processors and their basic configuration, see the official OpenTelemetry documentation.

Exporter

An exporter is a component that sends the processed data to one or more backends. Exporters also support more than one data source.

Because many exporters require additional configuration (for example, an endpoint), be sure to check the official OpenTelemetry documentation for a list of available exporters and their configurations.

Extensions optional

An extension is a component available for tasks that do not primarily concern processing telemetry data (for example, health monitoring and service discovery). By default, no extensions are configured.

You can configure the following extensions:

yamlcopydownload
extensions:
  health_check:
  pprof:
  zpages:
  memory_ballast:
    size_mib: 512

Services

This section is used to configure which components are enabled.

Important: Even if a component is properly configured in its rightful section, it will not be enabled unless it is defined within the service section.

yamlcopydownload
service:
  pipelines:
    metrics:
      receivers: [opencensus, prometheus]
      exporters: [opencensus, prometheus]
    traces:
      receivers: [opencensus, jaeger]
      processors: [batch]
      exporters: [opencensus, zipkin]

Within the service section, you need to define each component separately.

• Extensions can be enabled in their own section, while receivers, processors, and exporters are grouped under a pipeline section.

• Pipelines can be of type traces, metrics, or logs.

• Each receiver/processor/exporter can be used in more than one pipeline. For processors referenced in multiple pipelines, each pipeline gets a separate instance of the processors. This contrasts with receivers/exporters referenced in multiple pipelines, where only one instance of a receiver/exporter is used for all pipelines. Also, note that the order of processors dictates the order in which data is processed.

• You can also define the same components more than once. For example, you can have two different receivers or even two or more distinct parts of the pipeline.

Example configuration

Example of a standard collector configuration YAML file:

yamlcopydownload
receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  otlphttp:
    endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/otlp"
    headers:
      Authorization: "Api-Token <API_TOKEN>"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]

yamlcopydownload
receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  dynatrace:
    endpoint: "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest"
    api_token: "<API_TOKEN>"
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [dynatrace]

You can use environment variables in the configuration YAML file. This is useful, for example, for IDs or keys.

In both scenarios above, the exporters are specified in the pipelines section to enable them.
There are no extensions specified, since we will not be using any.

Advantages of using a collector

• The collector is vendor-agnostic, and helps to centralize and distribute traces, as it receives data from various sources at one point and can export them to various backend formats.
• The collector is especially important where a data source is not compatible with all endpoints.
• The collector configuration is in one YAML file, which eliminates the need to browse through multiple files in potentially differing languages to find the exact setting you are looking for, and means there is less code to maintain.
• The collector supports tail-based sampling, meaning the sampling decision is delayed until all spans of the trace are collected. The role of the collector here is buffering and grouping spans by trace ID.
• The collector also supports attribute processing, enabling the application of changes on resource attributes; this can be configured in the processors section.
• The collector is relatively lightweight, so teams can deploy their own collectors to avoid everyone working on the same configuration.
• The collector is easily configurable to send data to Dynatrace.