How do I monitor Kubernetes?

Based on Docker, Kubernetes is a next generation platform for deploying and running containerized applications at scale.

To monitor applications running within a Kubernetes cluster, install Dynatrace OneAgent on each cluster node, as described below.

Locate your Dynatrace OneAgent installer URL

If you plan to install Dynatrace OneAgent on more than 50 hosts, please consider serving the installer script via a dedicated server, such as Amazon S3. Otherwise, with more than 50 concurrent connections, Dynatrace Server may throttle requests.

The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you during Dynatrace OneAgent installation.

To get your ONEAGENT_INSTALLER_SCRIPT_URL

  1. Select Deploy Dynatrace from the navigation menu.
  2. Click Start installation and select Linux.
  3. Locate your ONEAGENT_INSTALLER_SCRIPT_URL, as shown below.

Installation

The following DaemonSet installs Dynatrace OneAgent for full-stack monitoring on each node of a Kubernetes cluster.

$ cat <<- EOF > dynatrace-oneagent.yml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: dynatrace-oneagent
spec:
  template:
    metadata:
      labels:
        name: dynatrace-oneagent
    spec:
      hostPID: true
      hostIPC: true
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/os: linux
      volumes:
      - name: host-root
        hostPath:
          path: /
      containers:
      - name: dynatrace-oneagent
        image: dynatrace/oneagent
        env:
        - name: ONEAGENT_INSTALLER_SCRIPT_URL
          value: "REPLACE_WITH_YOUR_URL"
        - name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
          value: "false"
        volumeMounts:
        - name: host-root
          mountPath: /mnt/root
        securityContext:
          privileged: true
EOF

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml file. Be sure to replace REPLACE_WITH_YOUR_URL with the Dynatrace OneAgent installer URL as explained above:

$ kubectl create -f dynatrace-oneagent.yml --namespace=kube-system
daemonset "dynatrace-oneagent" created

Verify that the dynatrace-oneagent daemon set has been created successfully:

$ kubectl get pods --namespace=kube-system
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m
$ kubectl logs -f dynatrace-oneagent-abcde
09:46:18 Deploying agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp
...
09:46:24 Validating agent installer in /tmp/Dynatrace-OneAgent-Linux.sh
Verification successful
09:46:24 Started agent deployment as docker image, PID 1234.
09:46:24 Container version: 1.x
09:46:24 Checking root privileges...
09:46:24 OK
09:46:27 Installation started, version 1.x, build date: 01.01.2017, PID 1234.
...

Uninstallation

Uninstalling Dynatrace OneAgent from each Kubernetes node can be achieved as follows:

Delete the dynatrace-oneagent daemon set:

$ kubectl delete ds/dynatrace-oneagent --namespace=kube-system

Updating

Whenever a new version of Dynatrace OneAgent becomes available in Dynatrace, you can re-deploy Dynatrace OneAgent as explained in the steps below. Your dynatrace/oneagent image will automatically fetch the latest version of Dynatrace OneAgent. If you've specified a default OneAgent install version for new hosts and applications in your OneAgent updates settings, the dynatrace/oneagent image will automatically fetch the defined default version of Dynatrace OneAgent:

Delete the dynatrace-oneagent daemon set:

$ kubectl delete ds/dynatrace-oneagent

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml file. Be sure to replace REPLACE_WITH_YOUR_URL with the Dynatrace OneAgent installer URL as explained above:

$ kubectl create -f dynatrace-oneagent.yml --namespace=kube-system
daemonset "dynatrace-oneagent" created

Pitfalls

Find out how to solve common problems that you may encounter.

Deployment seems successful, the dynatrace/oneagent container does not show up as ready

$ kubectl get ds/dynatrace-oneagent --namespace=kube-system
NAME                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR                 AGE
dynatrace-oneagent   1         1         0         1            0           beta.kubernetes.io/os=linux   14m
$ kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
09:46:18 Started agent deployment as docker image, PID 1234.
09:46:18 Agent installer can only be downloaded from secure location. Your installer URL should start with 'https': REPLACE_WITH_YOUR_URL

Please replace REPLACE_WITH_YOUR_URL in dynatrace-oneagent.yml with the Dynatrace OneAgent installer URL as explained above.

Deployment seems successful, but the dynatrace-oneagent container does not produce meaningful logs

$ kubectl get pods --namespace=kube-system
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
$ kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container has not yet fully started. Simply wait a couple of seconds.

Deployment was successful, but monitoring data is not available in Dynatrace

$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs if application containers have started before Dynatrace OneAgent was fully installed on the system. As a consequence, some parts of your application run uninstrumented. To be on the safe side, Dynatrace OneAgent should be fully integrated before you start your application containers. If your application has already been running, restarting its containers will have the very same effect.