Monitor Kubernetes/OpenShift events
Prerequisites
- For Dynatrace Managed customers, Dynatrace Managed version 1.234+
- ActiveGate version 1.223+
- In Dynatrace, go to your Kubernetes cluster settings page and make sure that Monitor Kubernetes namespaces, services, workloads, and pods is turned on.
- Enable the latest version of Dynatrace log monitoring
Kubernetes events monitoring for analysis and alerting
For full observability into your Kubernetes events, automatic Davis® analysis, and custom alerting, you need to enable Kubernetes event monitoring.
To enable event monitoring for specific Kubernetes clusters,
- In the Dynatrace menu, go to Kubernetes.
- Find your Kubernetes cluster and select More (…) > Settings.
- Turn on Monitor events.
- Select Save changes.
When you enable Monitor events, all events are ingested and all important events are considered in the Davis root-cause detection. Alternatively, for maximum flexibility and fine-grained control over the events you want to ingest from Kubernetes, you can filter events.
Inferred Kubernetes events
Even if Monitor events is disabled, the so-called inferred Kubernetes events are still ingested. These inferred events aren't native Kubernetes events but are created by ActiveGate based on the information from the Kubernetes API server.
Examples of inferred events:
-
cgroup OOM kill events
-
Workload specification changes (replicas, images, environment variables, resources, probes) for
- Deployments
- StatefulSets
- DaemonSets
These events aren't billed and are relevant for Davis root-cause analysis.
View events
After enabling the Kubernetes event monitoring, you can view and analyze events from the Kubernetes cluster.
On your Kubernetes cluster details page, go to Events.
You can filter events by:
- Timeframe: select one of the timeframes in the chart to view open events for that timeframe
- Specific events: select one of the group labels below the chart to view specific events
For more information about an event, select Details for the event.
Note: Kubernetes events are associated with Kubernetes entities. An event is displayed on the respective entity page and on related entity pages. For example, pod events are displayed on the cluster, namespace, workload, and pod details page.
You can also view events on the Log viewer page (in the Dynatrace menu, go to Logs), which allows for advanced search and filtering.
Note: Inferred Kubernetes events are not shown in Log viewer. They are directly ingested as events.
Filter events to be monitored
Filtering is turned off by default, which means that all events are ingested. To set up monitoring only for certain events
-
Turn on Filter events.
If you don't see the Filter events switch, make sure that Monitor events is turned on first.
-
Set up multiple field selectors for every Kubernetes environment.
-
optional Have Davis perform root cause analysis on all important Kubernetes events.
-
Select Save changes.
Set up event field selectors
Filtering follows the Kubernetes-established syntax of field selectors, so events can be chosen based on event resource fields such as source.component
, type
, or involvedObject
.
A field selector expression must meet the following requirements:
-
It must conform to the following regular expression:
^[\w\.]{1,1024}((=){1,2}|(!=))[^,!=]{0,256}(,[\w\.]{1,1024}?((=){1,2}|(!=))[^,!=]{0,256}){0,9}$
. -
It can have up to 10 selectors separated by commas. Events matching all comma-separated selectors will be ingested. The logical operator is
AND
. -
A selector consists of three parts:
- Key: Contains up to 1,024 alphanumerical characters, underscores, and dots.
- Operator: Is either
=
,==
or!=
. - optional Value: Can contain up to 256 characters. Cannot contain exclamation marks, equal signs, or commas.
For example,
- If you set a field selector expression to
involvedObject.namespace=hipster-shop,type=Warning
, the expression will store all the events related to the namespacehipster-shop
that are of typeWarning
. - If you separate the expression into two independent field selectors, you'll get all events for namespace
hipster-shop
and all events of typeWarning
. The logical operator in this case isOR
.
Example event field selectors:
Event field selectors | Field selector expression |
---|---|
Get all Node events | involvedObject.kind=Node |
Get all Warning events | type=Warning |
Get all Pod events | involvedObject.kind=Pod |
Get all events of objects related to a specific namespace | involvedObject.namespace=<your_namespace> (Make sure to replace <your_namespace> with the name of your own namespace) |
Get all BackOff events for pods across all namespaces | reason=BackOff |
Get all events with non-empty type field | type!= |
Get all nginx container events | involvedObject.fieldPath==spec.containers{nginx} |
To set up event field selectors, select one of the options below:
- In the Dynatrace menu, go to Kubernetes.
- Find your cluster and select More (…) > Settings.
- Turn on Filter events.
- Select Add events field selector.
- Enter a Field selector name and Field selector expression.
- Select Save changes.
Example command:
kubectl get events --all-namespaces --field-selector involvedObject.namespace=hipster-shop,type=Warning
You can define the event field selectors via Dynatrace API.
Note: You can create a maximum of 20 event filter rules per Kubernetes cluster.
Monitor important events
When problems with applications, microservices, or infrastructure are detected, Davis performs root cause analysis on all important Kubernetes events for nodes, namespaces, workloads, and pods.
By default, all these events are monitored when Monitor events is turned on. If you choose to Filter events, then both, important events and events matching the filter condition, are ingested.
To enable monitoring of important events, when event filtering is turned on
-
In the Dynatrace menu, go to Kubernetes.
-
Find your cluster and select More (…) > Settings.
-
Turn on Include important events.
If you don't see the Include important events switch, make sure that Monitor events and Filter events are turned on first.
-
Select Save changes.
Charting and custom alerting for events
Charting and custom alerting for events allow you to trigger alerts whenever Kubernetes events occur (for example, always alert in case of a BackOff
event).
To trigger alerts for Kubernetes events:
-
Create a log event based on the events for which you want to chart or receive alerts. Be sure to select the Event Type as Custom Alert under the Event template.
-
Alternatively, you can create a log metric based on the events for which you want to chart or receive alerts. Be sure to add the dimensions you want for later use (for splitting or filtering). For example, if you need to chart or configure an alert only for a specific namespace, you need to add the namespace as a dimension of the log metric.
Note: New log metrics appear in Dynatrace only after the first matching event is ingested.
Then, use the log metric created in the previous step to either chart it using Data explorer or set up an alert using custom event for alerting.
Licensing
To estimate the number of events that consume DDUs, you can query the dsfm:active_gate.kubernetes.events.processed
metric, which provides information about the number of events that are being ingested into Dynatrace per Kubernetes cluster.
Example query for a 24-hour timeframe:
dsfm:active_gate.kubernetes.events.processed:splitBy("dt.entity.kubernetes_cluster"):sum:auto:sort(value(sum,descending)):limit(10)
Note: DDU consumption applies to Kubernetes event monitoring. For details, see DDUs for custom Davis events.