Monitor Kubernetes events

Prerequisites

• Dynatrace SaaS 1.224+
• ActiveGate version 1.223+
• Enable Log Monitoring v2
• Make sure monitoring is enabled on your cluster, and turn on Monitor events in the settings of your Kubernetes cluster.

Kubernetes events integration for analysis and alerting

For full observability into your Kubernetes events, automatic Davis analysis, and custom alerting, you need to enable Kubernetes event integration.

Note: DDU consumption applies to Kubernetes event integration. For details, see DDUs for custom Davis events.

You can enable this feature for all Kubernetes clusters or for specific Kubernetes clusters. See below for instructions.

Enable for all Kubernetes clusters

To enable event integration for all Kubernetes clusters

1. In the Dynatrace menu, go to Kubernetes.
2. Select Upgrade Kubernetes event integration.
3. Select Save changes.

Enable for specific Kubernetes clusters

To enable event integration for individual clusters

1. In the Dynatrace menu, go to Kubernetes.
2. Select your cluster.
3. Select Upgrade Kubernetes event integration.
4. Select Save changes.

Change monitoring settings

You can change monitoring settings at any time from your Kubernetes cluster details page.

1. In the Dynatrace menu, go to Kubernetes.
2. Find your Kubernetes cluster, and then select More (…) > Settings in the Actions column.
3. Change Opt in to the Kubernetes event integration for analysis and alerting as needed, and then select Save changes.

Note: You can't disable Kubernetes event integration for all Kubernetes clusters at the same time. You need to manually disable it for each cluster according to the above instructions.

View events

After enabling the Kubernetes event integration, you can view and analyze events from the Kubernetes cluster. On your Kubernetes cluster details page, go to Events.

You can filter events by

• Timeframe: select one of the timeframes in the chart to view open events for that specific timeframe
• Specific events: select one of the group labels below the chart to view specific events

For more information about an event, select Details for the respective event.

Note: Kubernetes events are associated with Kubernetes entities. An event is displayed on the respective entity page and on related entity pages. For example, pod events are displayed on the cluster, namespace, workload, and pod detail page.

Create log events

The ingestion of Kubernetes events is a two-step process:

Step 1: ActiveGate forwards the Kubernetes events to the Dynatrace Server, where they are stored as log lines in the Dynatrace log storage.

Step 2: Selected log lines, configured in the log events configuration file, are ingested into the Dynatrace event system as event reports, and stored in the event storage to be processed as part of the problem detection.

By default, all Kubernetes log events are enabled, but you can also create your own log events. This enables you to

• Create specific metrics on Kubernetes events
• Raise notifications on certain event criteria

To create custom log events

1. In the Dynatrace menu, go to Settings.
2. Under Log Monitoring, select Log events.
3. Select Add log event.
4. Enter the requested information, and then select Save changes.

For details, see Log events and Log metrics.

Monitor Kubernetes events

Dynatrace provides a flexible way of ingesting Kubernetes events into your environment to enrich the existing monitoring data from OneAgents and ActiveGates with additional context information. Ingestion follows the Kubernetes-established format of field selectors, so events can be chosen based on event resource fields such as source.component, type, or involvedObject.

Set up event field selectors optional

Event filtering is turned off by default, which means that all events are ingested. For maximum flexibility and fine-grained control over the events you want to ingest from Kubernetes, you can set up multiple field selectors for every Kubernetes environment.

To set up field selectors

1. In the Dynatrace menu, go to Kubernetes.
2. Look for your cluster, and then select More (…) > Settings in the Actions column.
3. Turn on Filter events.
4. Select Add events field selector.
5. Select Save changes.

Note: You can create a maximum of 20 event filter rules per Kubernetes cluster.

Syntax

The field selector syntax is the same as the one used in Kubernetes. An event field selector expression can have up to 10 selectors separated by commas. Events matching all comma-separated selectors will be ingested. The logical operator is AND.

If you set a field selector expression to infolvedObject.namespace=hipster-shop,type=Warning, the expression will store all the events related to the namespace hipster-shop that are of type Warning.
If you separate the expression into two independent field selectors, you'll get all events for namespace hipster-shop and all events of type Warning. The logical operator in this case is OR.

Examples

CLI equivalent command

Alternatively, you can select an event field from the CLI as shown in the example below.

copydownload
kubectl get events --all-namespaces --field-selector involvedObject.namespace=hipster-shop,type=Warning

Monitor important events

When problems with applications, microservices, or infrastructure are detected, Davis performs root cause analysis on all important Kubernetes events for nodes, namespaces, workloads, and pods.

If you want Davis to look out for Kubernetes events during the root-cause analysis for an application issue, follow the steps below.

1. In the Dynatrace menu, go to Kubernetes.
2. Look for your cluster, and then select More (…) > Settings in the Actions column.
3. Turn on Filter events.
4. Turn on Include important events.
5. Select Save changes.

Charting and custom alerting for events

Charting and custom alerting for events allow you to trigger alerts whenever Kubernetes events occur (for example, always alert in case of a CrashLoopBackOff event).

To trigger alerts for Kubernetes events

1. Create a log metric based on the events for which you want to chart or receive alerts. Be sure to add the dimensions you want for later use (for splitting or filtering). For example, if you need to chart or configure an alert only for a specific namespace, you need to add the namespace as a dimension of the log metric.

Note: New log metrics appear in Dynatrace only after the first matching event is ingested.

2. Use the log metric created in the previous step to either chart it using our Data explorer or set up an alert using custom event for alerting.

Licensing

To estimate the number of events that consume DDUs, you can query the dsfm:active_gate.kubernetes.events.processed metric, which provides information about the number of events that are being ingested into Dynatrace per Kubernetes cluster.

Example query for a 24-hour timeframe:

dsfm:active_gate.kubernetes.events.processed:splitBy("dt.entity.kubernetes_cluster"):sum:auto:sort(value(sum,descending)):limit(10)

For information on licensing, see DDUs for custom Davis events.