• Home
  • Deploy
  • Kubernetes
  • Deployment
  • Troubleshooting
  • Monitoring issues troubleshooting

Monitoring issues troubleshooting

This guide provides general troubleshooting steps and guidance for common issues encountered when using Dynatrace with Kubernetes. It covers how to access debug logs, use the troubleshoot subcommand, or generate a support archive.

Pods stuck in Terminating state after upgrade

If your CSI driver and OneAgent pods get stuck in Terminating state after upgrading from Dynatrace Operator version 0.9.0, you need to manually delete the pods that are stuck.

Run the command below.

bash
kubectl delete pod -n dynatrace --selector=app.kubernetes.io/component=csi-driver,app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/version=0.9.0 --force --grace-period=0
bash
oc delete pod -n dynatrace --selector=app.kubernetes.io/component=csi-driver,app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/version=0.9.0 --force --grace-period=0

Unable to retrieve the complete list of server APIs

Dynatrace Operator

plaintext
unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

If the Dynatrace Operator pod logs this error, you need to identify and fix the problematic services. To identify them

  1. Check available resources.

    bash
    kubectl api-resources

  2. If the command returns this error, list all the API services and make sure there aren't any False services.

    bash
    kubectl get apiservice

CrashLoopBackOff: Downgrading OneAgent is not supported, please uninstall the old version first

Dynatrace Operator

If you get this error, the OneAgent version installed on your host is later than the version you're trying to run.

Solution: First uninstall OneAgent from the host, and then select your desired version in the Dynatrace web UI or in DynaKube. To uninstall OneAgent, connect to the host and run the uninstall.sh script. (The default location is /opt/dynatrace/oneagent/agent/uninstall.sh)

For CSI driver deployments, use the following commands instead:

  1. Delete the DynaKube custom resources.
  2. Delete the CSI driver manifest.
  3. Delete the /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com directory from all Kubernetes nodes.
  4. Reapply the CSI driver and DynaKube custom resources.

Crash loop on pods when installing OneAgent

Application-only monitoring

If you get a crash loop on the pods when you install OneAgent, you need to increase the CPU memory of the pods.

Deployment seems successful but the dynatrace-oneagent container doesn't show up as ready

DaemonSet

bash
kubectl get ds/dynatrace-oneagent --namespace=kube-system
NAME                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR                 AGE
dynatrace-oneagent   1         1         0         1            0           beta.kubernetes.io/os=linux   14mc
bash
kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
09:46:18 Started agent deployment as Docker image, PID 1234.
09:46:18 Agent installer can only be downloaded from secure location. Your installer URL should start with 'https': REPLACE_WITH_YOUR_URL

Change the value REPLACE_WITH_YOUR_URL in the dynatrace-oneagent.yml DaemonSet with the Dynatrace OneAgent installer URL.

bash
oc get pods

NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
bash
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account hasn't been allowed to pull images from the RHCC.

Deployment seems successful, however the dynatrace-oneagent image can't be pulled

DaemonSet

bash
oc get pods

NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
bash
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account hasn't been allowed to pull images from the RHCC.

Deployment seems successful, but the dynatrace-oneagent container doesn't produce meaningful logs

DaemonSet

bash
kubectl get pods --namespace=kube-system
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
bash
kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating
bash
oc get pods

NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
bash
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container hasn't yet fully started. Simply wait a few more seconds.

Deployment seems successful, but the dynatrace-oneagent container isn't running

DaemonSet

bash
oc process -f dynatrace-oneagent-template.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc apply -f -

daemonset "dynatrace-oneagent" created

Please note that quotes are needed to protect the special shell characters in the OneAgent installer URL.

bash
oc get pods

No resources found.

This is typically the case if the dynatrace service account hasn't been configured to run privileged pods.

bash
oc describe ds/dynatrace-oneagent

Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
FirstSeen LastSeen Count From SubObjectPath Type     Reason       Message
--------- -------- ----- ---- ------------- -------- ------------ -------
6m        3m       17         {daemon-set } Warning  FailedCreate Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]

Deployment was successful, but monitoring data isn't available in Dynatrace

DaemonSet

bash
kubectl get pods --namespace=kube-system
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m
bash
oc get pods

NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs if application containers have started before OneAgent was fully installed on the system. As a consequence, some parts of your application run uninstrumented. To be on the safe side, OneAgent should be fully integrated before you start your application containers. If your application has already been running, restarting its containers will have the very same effect.

No pods scheduled on control-plane nodes

DaemonSet

Kubernetes version 1.24+

Taints on master and control plane nodes are changed on Kubernetes versions 1.24+, and the OneAgent DaemonSet is missing appropriate tolerations in the DynaKube custom resource.

To add the necessary tolerations, edit the DynaKube YAML as follows.

yaml
tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

Error when applying the custom resource on GKE

plaintext
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.dynatrace.com": Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=2s (https://dynatrace-webhook.dynatrace.svc/validate?timeout=2s)": context deadline exceeded

If you are getting this error when trying to apply the custom resource on your GKE cluster, the firewall is blocking requests from the Kubernetes API to the Dynatrace Webhook because the required port (8443) is blocked by default.

The default allowed ports (443 and 10250) on GCP refer to the ports exposed by your nodes and pods, not the ports exposed by any Kubernetes services. For example, if the cluster control plane attempts to access a service on port 443 such as the Dynatrace webhook, but the service is implemented by a pod using port 8443, this is blocked by the firewall.

To fix this, add a firewall rule to explicitly allow ingress to port 8443.

For more information about this issue, see API request that triggers admission webhook timing out.

CannotPullContainerError

If you get errors like this on your pods when installing Dynatrace OneAgent, your Docker download rate limit has been exceeded.

CannotPullContainerError: inspect image has been retried [X] time(s): httpReaderSeeker: failed open: unexpected status code

For details, consult the Docker documentation.

Limit log timeframe

cloudNativeFullStack applicationMonitoring

Dynatrace Operator version 0.10.0+

If there's DiskPressure on your nodes, you can configure the CSI driver log garbage collection interval to lower the storage usage of the CSI driver. The default value of keeping logs before they are deleted from the file system is 7 (days). To edit this timeframe, select one of the options below, depending on your deployment mode.

Be careful when setting this value; you might need the logs to investigate problems.

  1. Edit the manifests of the CSI driver daemonset (kubernetes-csi.yaml, openshift-csi.yaml), by replacing the placeholders (<your_value>) with your value.

    yaml
    apiVersion: apps/v1
kind: DaemonSet
...
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
        ...
      - name: provisioner
      ...
        env:
          - name: MAX_UNMOUNTED_VOLUME_AGE
            value: <your_value> # defined in days, must be a plain number. `0` means logs are immediately deleted. If not set, defaults to `7`.

  2. Apply the changes.

Edit values.yaml to set the maxUnmountedVolumeAge parameter under the csidriver section.

yaml
csidriver:
  enabled: true
  ...
  maxUnmountedVolumeAge: "" # defined in days, must be a plain number. `0` means logs are immediately deleted. If not set, defaults to `7`.