Enrich from Kubernetes

The following configuration example shows how you configure a Collector instance to enrich OTLP request with additional Kubernetes metadata.

Demo configuration

yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
       node_from_env_var: K8S_NODE_NAME
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.pod.start_time
      # Pod labels which can be fetched via K8sattributeprocessor
      labels:
        - tag_name: key1
          key: label1
          from: pod
        - tag_name: key2
          key: label2
          from: pod
    # Pod association using resource attributes and connection
    pod_association:
      - sources:
         - from: resource_attribute
           name: k8s.pod.uid
      - sources:
         - from: resource_attribute
           name: k8s.pod.ip
      - sources:
         - from: connection

exporters:
  otlphttp:
    endpoint: $DT_ENDPOINT/api/v2/otlp
    headers:
      Authorization: "Api-Token $DT_API_TOKEN"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [k8sattributes]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [k8sattributes]
      exporters: [otlphttp]

Prerequisites

Contrib distribution or a custom Builder version with the Kubernetes Attributes processor
Collector deployed in agent mode
The API URL of your Dynatrace environment
An API token with the relevant access scope
Kubernetes configured for the required role-based access control

Kubernetes configuration

Configure the following rbac.yaml file with your Kubernetes instance, to allow the Collector to use the Kubernetes API with the service-account authentication type.

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: otelcontribcol
  name: otelcontribcol
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otelcontribcol
  labels:
    app: otelcontribcol
rules:
  - apiGroups:
      - ""
    resources:
      - events
      - namespaces
      - namespaces/status
      - nodes
      - nodes/spec
      - nodes/stats
      - nodes/proxy
      - pods
      - pods/status
      - replicationcontrollers
      - replicationcontrollers/status
      - resourcequotas
      - services
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "events.k8s.io"
    resources:
      - events
    verbs:
      - watch
  - apiGroups:
      - apps
    resources:
      - daemonsets
      - deployments
      - replicasets
      - statefulsets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - daemonsets
      - deployments
      - replicasets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - batch
    resources:
      - jobs
      - cronjobs
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - autoscaling
    resources:
      - horizontalpodautoscalers
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otelcontribcol
  labels:
    app: otelcontribcol
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otelcontribcol
subjects:
  - kind: ServiceAccount
    name: otelcontribcol
    namespace: default

Components

For our configuration, we configured the following components.

Receiver

Under receivers, we specify the standard otlp receiver as an active receiver component for our Collector instance.

This is mainly for demonstration purposes. You can specify any other valid receiver here (for example, zipkin).

Processor

Under processors, we specify the k8sattributes processor with the following parameters:

auth_type—Specifies to use a service account to obtain the necessary data.
passthrough—Set to false, in order not to forward IP address related information. Should be true when running as gateway.
filter—Specifies that data should be filtered by the node name in the default environment variable K8S_NODE_NAME.
extract—Specifies which information should be extracted.
pod_association—Specifies how the pod information is linked to attributes.

Exporter

Under exporters, we specify the default otlphttp exporter and configure it with our Dynatrace API URL and the required authentication token.

For this purpose, we set the following two environment variables and reference them in the configuration values for endpoint and Authorization.

DT_ENDPOINT contains the base URL of your ActiveGate (for example, https://{your-environment-id}.live.dynatrace.com)
DT_API_TOKEN contains the API token

Service pipeline

Under service, we assemble our receiver and exporter objects into pipelines for traces and metrics and specifically reference our Kubernetes processor under processors.