Log Monitoring in Kubernetes

Dynatrace Log Monitoring supports collecting logs from Kubernetes container orchestration systems via OneAgent.

How autodiscovery works

For Kubernetes version 1.14+, OneAgent autodiscovers log messages written to the containerized application's stdout/stderr streams. Logs written directly to pods are not discovered by OneAgent. Kubernetes Engine saves these log streams to a file in the /var/log/pods directory on the Kubernetes node. OneAgent autodiscovers these log files from that path.

Autodiscovery requirements

Requirements for autodiscovery of Kubernetes logs:

• Docker, containerd, or CRI-O container runtime is used.
• The process running in the container is an important process.
• Logs are written to the container's stdout/stderr streams.
• The log file in /var/log/pods exists for a minimum of one minute after container execution is finished.

Connecting logs with Kubernetes properties and annotations

Before you start, you need to deploy Dynatrace Operator and enable Kubernetes API monitoring.

Note: Only the classic full-stack and cloud-native full-stack injections are supported. For details, see Deployment options on Kubernetes/OpenShift.

As OneAgent discovers logs from /var/log/pods path on Kubernetes node, it connects specific logs to corresponding pods based on the pod uid from the file path.

In addition, OneAgent enhances these logs with the following Kubernetes metadata:

• dt.kubernetes.cluster.name
• dt.kubernetes.node.system_uuid
• k8s.pod.name
• k8s.pod.uid
• k8s.container.name
• k8s.namespace.name
• k8s.deployment.name
• dt.entity.container_group
• dt.entity.container_group_instance
• dt.entity.kubernetes_node
• dt.entity.cloud_application_instance

This metadata is used to map the logs to the entity model of Kubernetes clusters, namespaces, workloads, and pods. As a result, the logs are present in the Kubernetes entity model on the Dynatrace platform.

Add specific log sources to storage

Log addition to storage in Kubernetes is not automatic.

To select logs to include in storage or exclude from storage

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log sources and storage.

2. Select Include all logs, Include the following logs, or Exclude the following logs from the list.

   If you select Include the following logs or Exclude the following logs, two tabs are displayed:

   • Hosts perspective
   • Process groups perspective

3. Switch between the tabs to select logs that you want to include or exclude (depending on what you selected above).

   Important: Only log sources selected in the currently active (selected) perspective tab will be saved. Log sources selected in the other tab will be ignored.

4. Select Save changes.