Monitor Kubernetes events

Prerequisites

  • Dynatrace SaaS version 1.224+
  • ActiveGate version 1.223+
  • Log Monitoring 2.0 must be activated
  • In your Dynatrace environment, go to Settings and select Cloud and virtualization > Kubernetes, select your cluster, and turn on Enable monitoring and Monitor events.

How to monitor Kubernetes events

Which Kubernetes events can be ingested?

Dynatrace provides a flexible way of ingesting Kubernetes events into your environment to enrich the existing monitoring data from OneAgents and ActiveGates with additional context information. Ingestion follows the Kubernetes-established format of field selectors, so events can be chosen based on event resource fields such as source.component, type, or involvedObject.

Example:
hipster

Set up event field selectors

You can set up multiple field selectors for every Kubernetes environment to get maximum flexibility and fine-grained control over the events you want to ingest from Kubernetes.
Note: You must specify at least one field selector. You can create a maximum of 20 event filter rules per Kubernetes cluster.

You can also define the event field selectors via Dynatrace API.

Syntax

The field selector syntax is the same as the one used in Kubernetes. An event field selector expression can have up to 10 selectors separated by commas. Events matching all comma-separated selectors will be ingested. The logical operator is AND.

events-field-selector

The expression shown in the above example will store all the events related to the namespace hipster-shop that are of type Warning.
If you separate the expression into two independent field selectors, you'll get all events for namespace hipster-shop and all events of type Warning. The logical operator in this case is OR.

Examples

Events field selectors Field selector expression
Get all Node events involvedObject.kind=Node
Get all Warning events type=Warning
Get all Pod events involvedObject.kind=Pod
Get all events of objects related to a specific namespace involvedObject.namespace=<your_namespace> (Make sure to replace <your_namespace> with the name of your own namespace)
Get all BackOff events for pods across all namespaces reason=BackOff

CLI equivalent command

Alternatively, you can select an event field from the CLI as shown in the example below.

kubectl get events --all-namespaces --field-selector involvedObject.namespace=hipster-shop,type=Warning

Monitor important events

Dynatrace version 1.214+

Davis analyzes all important Kubernetes events for nodes, namespaces, workloads, pods, and containers in the root cause analysis when applications, microservices or infrastructure problems occur. Enable Include all events relevant for Davis in your Dynatrace environment to make sure Dynatrace integrates the Kubernetes events for Davis analytics.

Custom alerting for events

Dynatrace SaaS version 1.224+

To configure alerts for Kubernetes events

  1. Create a log metric based on the events for which you want to receive alerts. Be sure to add the dimensions you want for later use (for splitting or filtering). For example, if you need to configure an alert only for a specific namespace, you need to add the namespace as a dimension of the log metric.

Note: New log metrics appear in Dynatrace only after the first matching event is ingested.

  1. Configure a custom event for alerting based on the log metric created in the previous step.

Licensing

To estimate the number of events that consume DDUs, you can query the dsfm:active_gate.kubernetes.events.processed metric, which provides information about the number of events that are being ingested into Dynatrace per Kubernetes cluster.

Example query for a 24-hour timeframe:

dsfm:active_gate.kubernetes.events.processed:splitBy("dt.entity.kubernetes_cluster"):sum:auto:sort(value(sum,descending)):limit(10)

For information on licensing, see DDUs for custom Davis events.