General troubleshooting
This guide provides general troubleshooting steps and guidance for common issues encountered when using Dynatrace with Kubernetes. It covers how to access debug logs, use the troubleshoot
subcommand, or generate a support archive.
Troubleshoot common Dynatrace Operator setup issues using the troubleshoot
subcommand
Dynatrace Operator version 0.9.0+
Run the command below to retrieve a basic output on DynaKube status, such as:
-
Namespace: If the
dynatrace
namespace exists (name can be overwritten via parameter) -
DynaKube:
- If
CustomResourceDefinition
exists - If
CustomResource
with the given name exists (name can be overwritten via parameter) - If the API URL ends with
/api
- If the secret name is the same as DynaKube (or
.spec.tokens
if used) - If the secret has
apiToken
andpaasToken
set - If the secret for
customPullSecret
is defined
- If
-
Environment: If your environment is reachable from the Dynatrace Operator pod using the same parameters as the Dynatrace Operator binary (such as proxy and certificate).
-
OneAgent and ActiveGate image: If the registry is accessible; if the image is accessible from the Dynatrace Operator pod using the registry from the environment with (custom) pull secret.
kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot
If you use a different DynaKube name, add the --dynakube <your_dynakube_name>
argument to the command.
Example output if there are no errors for the above-mentioned fields:
{"level":"info","ts":"2022-09-12T08:45:21.437Z","logger":"dynatrace-operator-version","msg":"dynatrace-operator","version":"<operator version>","gitCommit":"<commithash>","buildDate":"<release date>","goVersion":"<go version>","platform":"<platform>"}
[namespace ] --- checking if namespace 'dynatrace' exists ...
[namespace ] √ using namespace 'dynatrace'
[dynakube ] --- checking if 'dynatrace:dynakube' Dynakube is configured correctly
[dynakube ] CRD for Dynakube exists
[dynakube ] using 'dynatrace:dynakube' Dynakube
[dynakube ] checking if api url is valid
[dynakube ] api url is valid
[dynakube ] checking if secret is valid
[dynakube ] 'dynatrace:dynakube' secret exists
[dynakube ] secret token 'apiToken' exists
[dynakube ] customPullSecret not used
[dynakube ] pull secret 'dynatrace:dynakube-pull-secret' exists
[dynakube ] secret token '.dockerconfigjson' exists
[dynakube ] proxy secret not used
[dynakube ] √ 'dynatrace:dynakube' Dynakube is valid
[dtcluster ] --- checking if tenant is accessible ...
[dtcluster ] √ tenant is accessible
Debug logs
By default, OneAgent logs are located in /var/log/dynatrace/oneagent
.
To debug Dynatrace Operator issues, run
kubectl -n dynatrace logs -f deployment/dynatrace-operator
oc -n dynatrace logs -f deployment/dynatrace-operator
You might also want to check the logs from OneAgent pods deployed through Dynatrace Operator.
kubectl get pods -n dynatrace
NAME READY STATUS RESTARTS AGE
dynatrace-operator-64865586d4-nk5ng 1/1 Running 0 1d
dynakube-oneagent-<id> 1/1 Running 0 22h
kubectl logs dynakube-oneagent-<id> -n dynatrace
oc get pods -n dynatrace
NAME READY STATUS RESTARTS AGE
dynatrace-operator-64865586d4-nk5ng 1/1 Running 0 1d
dynakube-classic-8r2kq 1/1 Running 0 22h
oc logs oneagent-66qgb -n dynatrace
Generate a support archive using the support-archive
subcommand
Dynatrace Operator version 0.11.0+
Use support-archive
to generate a support archive containing all the files that can be potentially useful for the RFA analysis:
operator-version.txt
—a file containing the current Operator version informationlogs
—logs from all containers of the Dynatrace Operator pods in the Dynatrace Operator namespace (usuallydynatrace
); this also includes logs of previous containers, if available:dynatrace-operator
dynatrace-webhook
dynatrace-oneagent-csi-driver
manifests
—the Kubernetes manifests for Dynatrace Operator components and deployed DynaKubes in the Dynatrace Operator namespacetroubleshoot.txt
—output of a troubleshooting command that is automatically executed by thesupport-archive
subcommandsupportarchive_console.log
—complete output of thesupport-archive
subcommand
Usage
To create a support archive, execute the following command.
kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive
The collected files are stored in a zipped tarball and can be downloaded from the pod using the kubectl cp
command.
kubectl -n dynatrace cp <operator pod name>:/tmp/dynatrace-operator/operator-support-archive.tgz ./tmp/dynatrace-operator/operator-support-archive.tgz
The recommended approach is to use the --stdout
parameter line switch to stream the tarball directly to your disk.
kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive --stdout > operator-support-archive.tgz
If you use the --stdout
parameter, all support archive command output is written to stderr
so as not to corrupt the support archive tar file.
Sample output
The following is sample output from running support-archive
with the --stdout
parameter.
kubectl exec -n dynatrace deployment/dynatrace-operator -- dynatrace-operator support-archive --stdout > operator-support-archive.tgz
[support-archive] dynatrace-operator {"version": "v0.11.0", "gitCommit": "...", "buildDate": "...", "goVersion": "...", "platform": "linux/amd64"}
[support-archive] Storing operator version into operator-version.txt
[support-archive] Starting log collection
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/server.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/provisioner.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/registrar.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-bdnpc/liveness-probe.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/server.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/provisioner.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/registrar.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-cb4pc/liveness-probe.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/server.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/provisioner.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/registrar.log
[support-archive] Successfully collected logs logs/dynatrace-oneagent-csi-driver-k8bl5/liveness-probe.log
[support-archive] Successfully collected logs logs/dynatrace-operator-6d9fd9b9fc-sw5ll/dynatrace-operator.log
[support-archive] Successfully collected logs logs/dynatrace-webhook-7d84599455-bfkmp/webhook.log
[support-archive] Successfully collected logs logs/dynatrace-webhook-7d84599455-vhkrh/webhook.log
[support-archive] Starting K8S object collection
[support-archive] Collected manifest for manifests/injected_namespaces/Namespace-default.yaml
[support-archive] Collected manifest for manifests/dynatrace/Namespace-dynatrace.yaml
[support-archive] Collected manifest for manifests/dynatrace/Deployment-dynatrace-operator.yaml
[support-archive] Collected manifest for manifests/dynatrace/Deployment-dynatrace-webhook.yaml
[support-archive] Collected manifest for manifests/dynatrace/StatefulSet-dynakube-activegate.yaml
[support-archive] Collected manifest for manifests/dynatrace/DaemonSet-dynakube-oneagent.yaml
[support-archive] Collected manifest for manifests/dynatrace/DaemonSet-dynatrace-oneagent-csi-driver.yaml
[support-archive] Collected manifest for manifests/dynatrace/DynaKube-dynakube.yaml
Debug configuration and monitoring issues using the Kubernetes Monitoring Statistics extension
The Kubernetes Monitoring Statistics extension can help you:
- Troubleshoot your Kubernetes monitoring setup
- Troubleshoot your Prometheus integration setup
- Get detailed insights into queries from Dynatrace to the Kubernetes API
- Receive alerts when your Kubernetes monitoring setup experiences issues
- Get alerted on slow response times of your Kubernetes API
Potential issues when changing the monitoring mode
- Changing the monitoring mode from
classicFullStack
tocloudNativeFullStack
affects the host ID calculations for monitored hosts, leading to new IDs being assigned and no connection between old and new entities. - If you want to change the monitoring method from
applicationMonitoring
orcloudNativeFullstack
toclassicFullstack
orhostMonitoring
, you need to restart all the pods that were previously instrumented withapplicationMonitoring
orcloudNativeFullstack
.