Troubleshoot issues on Kubernetes/OpenShift

Dynatrace Operator

Example error:

unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request

If the Dynatrace Operator pod logs this error, you need to identify and fix the problematic services. To identify them

1. Check available resources.

bashcopydownload
kubectl api-resources

2. If the command returns this error, list all the API services and make sure there aren't any False services.

bashcopydownload
kubectl get apiservice

Dynatrace Operator

If you get this error, the OneAgent version installed on your host is later than the version you're trying to run.

Solution: First uninstall OneAgent from the host, and then select your desired version in the Dynatrace web UI or in DynaKube. To uninstall OneAgent, connect to the host and run the uninstall.sh script. (The default location is /opt/dynatrace/oneagent/agent/uninstall.sh)

Note: For CSI driver deployments, use the following commands instead:

1. Delete the Dynakube custom resources.
2. Delete the CSI driver manifest.
3. Delete the /var/lib/kubelet/plugins/csi.oneagent.dynatrace.com directory from all Kubernetes nodes.
4. Reapply the CSI driver and DynaKube custom resources.

Application-only monitoring

If you get a crash loop on the pods when you install OneAgent, you need to increase the CPU memory of the pods.

DaemonSet

bashcopydownload
kubectl get ds/dynatrace-oneagent --namespace=kube-system
NAME                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR                 AGE
dynatrace-oneagent   1         1         0         1            0           beta.kubernetes.io/os=linux   14mc

bashcopydownload
kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
09:46:18 Started agent deployment as Docker image, PID 1234.
09:46:18 Agent installer can only be downloaded from secure location. Your installer URL should start with 'https': REPLACE_WITH_YOUR_URL

bashcopydownload
oc get pods

NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s

bashcopydownload
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

DaemonSet

Example error:

bashcopydownload
oc get pods

NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s

bashcopydownload
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account hasn't been allowed to pull images from the RHCC.

DaemonSet

Example error:

bashcopydownload
kubectl get pods --namespace=kube-system
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s

bashcopydownload
kubectl logs -f dynatrace-oneagent-abcde --namespace=kube-system
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

bashcopydownload
oc get pods

NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s

bashcopydownload
oc logs -f dynatrace-oneagent-abcde

Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container hasn't yet fully started. Simply wait a few more seconds.

DaemonSet

bashcopydownload
oc process -f dynatrace-oneagent-template.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc apply -f -

daemonset "dynatrace-oneagent" created

Please note that quotes are needed to protect the special shell characters in the OneAgent installer URL.

bashcopydownload
oc get pods

No resources found.

This is typically the case if the dynatrace service account hasn't been configured to run privileged pods.

bashcopydownload
oc describe ds/dynatrace-oneagent

Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
FirstSeen LastSeen Count From SubObjectPath Type     Reason       Message
--------- -------- ----- ---- ------------- -------- ------------ -------
6m        3m       17         {daemon-set } Warning  FailedCreate Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]

DaemonSet

Example:

bashcopydownload
kubectl get pods --namespace=kube-system
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

bashcopydownload
oc get pods

NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs if application containers have started before OneAgent was fully installed on the system. As a consequence, some parts of your application run uninstrumented. To be on the safe side, OneAgent should be fully integrated before you start your application containers. If your application has already been running, restarting its containers will have the very same effect.

DaemonSet

Kubernetes version 1.24+

Taints on master and control plane nodes are changed on Kubernetes versions 1.24+, and the OneAgent DaemonSet is missing appropriate tolerations in the DynaKube custom resource.

To add the necessary tolerations, edit the DynaKube YAML as follows.

yamlcopydownload
tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists

Example error:

plaintextcopydownload
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.dynatrace.com": Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=2s (https://dynatrace-webhook.dynatrace.svc/validate?timeout=2s)": context deadline exceeded

If you are getting this error when trying to apply the custom resource on your GKE cluster, the firewall is blocking requests from the Kubernetes API to the Dynatrace Webhook because the required port (8443) is blocked by default.

The default allowed ports (443 and 10250) on GCP refer to the ports exposed by your nodes and pods, not the ports exposed by any Kubernetes services. For example, if the cluster control plane attempts to access a service on port 443 such as the Dynatrace webhook, but the service is implemented by a pod using port 8443, this is blocked by the firewall.

To fix this, add a firewall rule to explicitly allow ingress to port 8443.

For more information about this issue, see API request that triggers admission webhook timing out.

If you get errors like this on your pods when installing Dynatrace OneAgent, your Docker download rate limit has been exceeded.

CannotPullContainerError: inspect image has been retried [X] time(s): httpReaderSeeker: failed open: unexpected status code

For details, consult the Docker documentation.

Example error on the ActiveGate deployment status page:

Problem with ActiveGate token (reason:Absent)

Example error on Dynatrace Operator logs:

bashcopydownload
{"level":"info","ts":"2022-09-22T06:49:17.351Z","logger":"dynakube-controller","msg":"reconciling DynaKube","namespace":"dynatrace","name":"dynakube"}
{"level":"info","ts":"2022-09-22T06:49:17.502Z","logger":"dynakube-controller","msg":"problem with token detected","dynakube":"dynakube","token":"APIToken","msg":"Token on secret dynatrace:dynakube missing scopes [activeGateTokenManagement.create]"}

Example error on DynaKube status:

bashcopydownload
status:
    ...
    conditions:
    - message: Token on secret dynatrace:dynakube missing scopes [activeGateTokenManagement.create]
      reason: TokenScopeMissing
      status: "False"
      type: APIToken

Starting Dynatrace Operator version 0.9.0, Dynatrace Operator handles the ActiveGate token by default. If you're getting one of these errors, follow the instructions below, according to your Dynatrace Operator version.

• For Dynatrace Operator versions earlier than 0.7.0: you need to upgrade to the latest Dynatrace Operator version.

• For Dynatrace Operator version 0.7.0 or later, but earlier than version 0.9.0:

  1. Create a new API token and enable the following permissions:

     • Access problem and event feed, metrics, and topology (DataExport)
     • PaaS integration - Installer download (InstallerDownload)
     • Create ActiveGate tokens (activeGateTokenManagement.create)

  2. Add the following annotation to the DynaKube custom resource.

     yamlcopydownload
     apiVersion: dynatrace.com/v1beta1
     kind: DynaKube
     metadata:
        name: dynakube
        namespace: dynatrace
        annotations:
           feature.dynatrace.com/enable-activegate-authtoken: "true"

The underlying host's container runtime doesn't contain the certificate presented by your endpoint.

Note: The skipCertCheck field in the DynaKube YAML does not control this certificate check.

Example error (the error message may vary):

plaintextcopydownload
desc = failed to pull and unpack image "<environment>/linux/activegate:latest": failed to resolve reference "<environment>/linux/activegate:latest": failed to do request: Head "<environment>/linux/activegate/manifests/latest": x509: certificate signed by unknown authority
Warning Failed ... Error: ErrImagePull
Normal BackOff ... Back-off pulling image "<environment>/linux/activegate:latest"
Warning Failed ... Error: ImagePullBackOff

In this example, if the description on your pod shows x509: certificate signed by unknown authority, you must fix the certificates on your Kubernetes hosts, or use the private repository configuration to store the images.

The certificate for the communication is invalid or expired. If you're using a self-signed certificate, check the mitigation procedures for the ActiveGate.

The bearer token is invalid and the request has been rejected by the Kubernetes API. Verify the bearer token. Make sure it doesn't contain any whitespaces. If you're connecting to a Kubernetes cluster API via a centralized external role-based access control (RBAC), consult the documentation of the Kubernetes cluster manager. For Rancher, see the guidelines on the official Rancher website.

There is already a request pending for this integration with an ActiveGate. Wait for a couple minutes and check back.

If you get this error after applying the DynaKube custom resource, your Kubernetes API server may be configured with a proxy. You need to exclude https://dynatrace-webhook.dynatrace.svc from that proxy.

cloudNativeFullStack applicationMonitoring Example error in the logs on the OneAgent pods: Initial connect: not successful - retrying after xs. You can fix this problem by increasing the OneAgent timeout. Add the following feature flag to DynaKube: Note: Be sure to replace the placeholder (<...>) with the name of your DynaKube custom resource.

bashcopydownload
kubectl annotate dynakube <name-of-your-DynaKube> feature.dynatrace.com/oneagent-initial-connect-retry-ms=6000  -n dynatrace

If you use Calico to handle or restrict network connections, you might experience connectivity issues, such as:

• The operator, webhook, and CSI driver pods are constantly restarting
• The operator cannot reach the API
• The CSI driver fails to download OneAgent
• Injection into pods doesn't work

If you experience these or similar problems, use our GitHub sample policies for common problems.

Notes:

• For the activegate-policy.yaml and dynatrace-policies.yaml policies, if Dynatrace Operator isn't installed in the dynatrace namespace (Kubernetes) or project (OpenShift), you need to adapt the metadata and namespace properties in the YAML files accordingly.
• The purpose of the agent-policy.yaml and agent-policy-external-only.yaml policies is to let OneAgents that are injected into pods open external connections. Only agent-policy-external-only.yaml is required, while agent-policy.yaml allows internal connections to be made, such as pod-to-pod connections, where needed.
• Because these policies are needed for all pods where OneAgent injects, you also need to adapt the podSelector property of the YAML files.