Monitor Prometheus metrics

Prometheus is an open-source monitoring and alerting toolkit which is popular in the Kubernetes community. Prometheus scrapes metrics from a number of HTTP(s) endpoints that expose metrics in the OpenMetrics format.

Dynatrace integrates any metrics from Prometheus exporters in K8s and makes them available for charting, alerting, and analysis. See the list of available exporters in the Prometheus documentation.

Prerequisites

  • ActiveGate version 1.201+ and Dynatrace version 1.202+
  • From the navigation menu, select Settings > Cloud and virtualization, edit a Kubernetes cluster and enable Monitor Prometheus exporters.
  • Annotated pod definitions, see below.

Annotate Prometheus exporter pods

Dynatrace collects metrics from any pods that are annotated with a metrics.dynatrace.com/scrape property set to true in the pod definition.

Depending on the actual exporter in a pod, you might need to set additional annotations to the pod definition in order to allow Dynatrace to properly ingest those metrics.

Enable metrics scraping required

Set metrics.dynatrace.com/scrape to 'true' to enable Dynatrace to collect Prometheus metrics exposed for this pod.

Path to metrics endpoint optional

Use metrics.dynatrace.com/path to override the default (/metrics) Prometheus endpoint.

Metrics port optional

By default, Prometheus metrics are available at the first exposed TCP port of the pod. We recommend to set metrics.dynatrace.com/port to respective port.

HTTP/HTTPS optional

Set metrics.dynatrace.com/secure to true if you want to collect metrics that are exposed by an exporter via HTTPS. The default value is false, because most exporters expose their metrics via HTTP.

Filter metrics optional

Use metric.dynatrace.com/filter to define a filter that allows you to either include ("mode": "include") or exclude (("mode": "exclude")) a list of metrics. If no filter annotation is defined, all metrics are collected.

See an example of a simple pod definition with the annotations.

apiVersion: v1
    kind: Pod
    metadata:
      name: mypod
      annotations:
        metrics.dynatrace.com/scrape: 'true'
        metrics.dynatrace.com/path: '/path/to-metrics'
        metrics.dynatrace.com/port: '9001'
        metrics.dynatrace.com/secure: 'false'
        metric.dynatrace.com/filter: |
        {
          "mode": "include",
          "names": [
              "redis_db_keys",
              "redis_db_values"
              ]
        }
    spec:
      containers:
      - name: mycontainer
        image: myregistry/myimage:mytag

View metrics on a dashboard

Metrics from Prometheus exporters are available in the Data Explorer for custom charting. Select Create custom chart and select Try it out in the top banner. For more information, see Explorer.

You can simply search for metric keys of all available metrics and define how you’d like to analyze and chart your metrics. After that you can pin your charts on a dashboard.

Metric alerts

You can also create custom alerts based on the Prometheus scraped metrics. Form the navigation menu, select Settings > Anomaly detection > Custom events for alerting and select Create custom event for alerting. In the Create custom event for alerting page, search for a Prometheus metric using its key and define your alert. For more information, see Metric events for alerting.

Limitations

The current limitations of the Prometheus metrics integration are as follows.

  • Scraping metrics from annotated pods only. Scraping metrics from K8s services or endpoints will come in a later release.
  • If you run multiple exporters in a pod, you need to set the metrics.dynatrace.com/port annotation to direct Dynatrace which one to use.
  • Up to 1000 pods with 200 metric data points each, per minute.