Get started with Kubernetes/OpenShift monitoring
This page describes how to set up monitoring via Dynatrace Operator version 0.8.0+ on Kubernetes (with kubectl
) and OpenShift (with oc
) using the following deployment options:
- Classic full-stack
- Cloud-native full-stack
- Host monitoring
- Automatic application-only
-
If you already set up monitoring using an earlier version of Dynatrace Operator, we recommend that you upgrade to the latest Dynatrace Operator version.
-
If you already set up monitoring using OneAgent Operator, please see the instructions for migrating to Dynatrace Operator, as the OneAgent Operator procedure is deprecated.
- For information about the Dynatrace Operator version changes, see Understand and configure the DynaKube custom resource.
There are two ways to set up Dynatrace Operator to monitor your Kubernetes cluster:
- Automated: The automated mode provides a simple setup with basic configuration using the Dynatrace web UI.
-
Manual: The manual mode allows complex configuration options.
Instructions for manual mode are below.
Prerequisites
- See Support lifecycle for Kubernetes/OpenShift for supported versions and minimum version requirements.
- Pods must allow egress to your Dynatrace environment or to your Environment ActiveGate in order for metric routing to work properly.
- For OpenShift (
cloudNativeFullStack
andapplicationMonitoring
with CSI driver deployments), you need to configure security context constraints (OpenShift). - For OpenShift Dedicated, you need the cluster-admin role.
Note: By default, the OneAgent and ActiveGate images are pulled from the configured Dynatrace Cluster endpoint, unless differently specified. For details, see Parameters.
Tokens and permissions required
You have two options for configuring tokens and permissions:
-
recommended In Dynatrace, go to Kubernetes > Connect automatically via Dynatrace Operator and quickly generate tokens with the necessary permissions.
-
For greater control over the configuration options, follow the instructions below.
-
Create a Dynatrace Operator token (former API token) and enable the following permissions.
-
optional For
cloudNativeFullStack
andapplicationMonitoring
deployments, if you want to enable metadata metric enrichment, create a data ingest token and enable the following permissions:- Ingest metrics (
metrics.ingest
) - Ingest logs (
logs.ingest
) - Ingest events (
events.ingest
)
- Ingest metrics (
1Creates an authentication token for your ActiveGate to connect to the Dynatrace Cluster. The token is rotated by Dynatrace Operator every 30 days. When an authentication token is rotated, the affected ActiveGate is automatically deleted and redeployed.
2All three scopes are needed if you want Dynatrace Operator to automatically handle the connection to ActiveGate via public API for Kubernetes monitoring.
-
Set up monitoring via Dynatrace Operator - manual mode
Create a dynatrace
namespace (Kubernetes)/Add a dynatrace
project (OpenShift)
Install Dynatrace Operator
Wait for Dynatrace Operator components to finish initialization
Create a secret holding your tokens
Download a preconfigured DynaKube custom resource sample
Review the parameters
Review the available configuration options
Apply the DynaKube custom resource
Verify the Dynatrace Operator image signature
Create a dynatrace
namespace (Kubernetes)/Add a dynatrace
project (OpenShift)
Depending on your platform, select one of the options below.
kubectl create namespace dynatrace
oc adm new-project --node-selector="" dynatrace
Install Dynatrace Operator
To install Dynatrace Operator, select one of the following, depending on your deployment type.
-
For
classicFullStack
,applicationMonitoring
without CSI driver, andhostMonitoring
when readonly is disabled, run the command below.kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml
oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml
-
For
cloudNativeFullStack
,applicationMonitoring
with CSI driver, andhostMonitoring
when readonly isn't disabled, run the commands below.kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes-csi.yaml
oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift-csi.yaml
Wait for Dynatrace Operator components to finish initialization
Run the following command to see when Dynatrace Operator components finish initialization.
kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300s
oc -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300s
Create a secret holding your tokens
To create a secret holding your tokens, select one of the following, depending on your deployment type.
-
For
classicFullStack
, create a secret nameddynakube
holding the API token obtained in Tokens and permissions required. Be sure to replace the placeholder (<...>
) with your own value.kubectl -n dynatrace create secret generic dynakube --from-literal="apiToken=<API_TOKEN>"
oc -n dynatrace create secret generic dynakube --from-literal="apiToken=<API_TOKEN>"
-
For
cloudNativeFullStack
andapplicationMonitoring
, create a secret nameddynakube
holding both the API token and thedataIngestToken
, see Tokens and permissions required. Be sure to replace the placeholders (<...>
) with your own values.kubectl -n dynatrace create secret generic dynakube --from-literal="apiToken=<API_TOKEN>" --from-literal="dataIngestToken=<DATA_INGEST_TOKEN>"
oc -n dynatrace create secret generic dynakube --from-literal="apiToken=<API_TOKEN>" --from-literal="dataIngestToken=<DATA_INGEST_TOKEN>"
Download a preconfigured DynaKube custom resource sample
Download one of the preconfigured DynaKube custom resource samples from GitHub, according to your monitoring approach.
Review the parameters
Review the available parameters, and adapt the DynaKube custom resource according to your requirements.
Review the available configuration options optional
Review the available configuration options for this integration:
- Configure build label propagation
- Add a custom properties file
- Add a custom certificate for ActiveGate
- Configure proxy
- Read-only file systems support
- Configure monitoring for namespaces and pods
- Import Kubernetes API certificates
- Configure security context constraints (OpenShift)
- Metadata metric enrichment
- Enable AppArmor for enhanced security
- High availability
- Using
priorityClass
for critical Dynatrace components - Set namespace-based isolation levels for pods
- Configure
customPullSecret
in DynaKube
Apply the DynaKube custom resource
Run the command below to apply the DynaKube custom resource, making sure to replace <your-DynaKube-CR>
with your actual DynaKube custom resource file name. A validation webhook will provide helpful error messages if there's a problem.
kubectl apply -f <your-DynaKube-CR>.yaml
oc apply -f <your-DynaKube-CR>.yaml
Verify the Dynatrace Operator image signature optional
Dynatrace Operator version 0.9.0+
To verify the signature of a Dynatrace Operator image
-
Select one of the following options:
Run the command below, making sure to replace
<version>
with your Dynatrace Operator version (for examplev0.10.4
).cosign verify --key https://github.com/Dynatrace/dynatrace-operator/releases/download/<version>/cosign.pub dynatrace/dynatrace-operator:<version>
Run the command below, making sure to replace
<version>
with your Dynatrace Operator version (for examplev0.10.4
).COSIGN_EXPERIMENTAL=1 cosign verify dynatrace/dynatrace-operator:<version>
Limitations
Dynatrace Operator doesn't support OpenShift version 3.11 or earlier.
Monitor large Kubernetes environments
Contact Dynatrace ONE if you want to monitor environments that are larger than:
- 1,000 Kubernetes clusters per Dynatrace environment
- 500 nodes per Kubernetes cluster
- 50,000 pods per Kubernetes cluster
Update Dynatrace Operator
To update Dynatrace Operator, select one of the following, depending on your deployment type.
-
For
classicFullStack
,applicationMonitoring
without CSI driver, andhostMonitoring
when readonly is disabled, run the command below.kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml
oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml
-
For
cloudNativeFullStack
,applicationMonitoring
with CSI driver, andhostMonitoring
when readonly isn't disabled, run the commands below.kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml kubectl apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes-csi.yaml
oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml oc apply -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift-csi.yaml
Dynatrace Operator version 0.9.0+
Starting with Dynatrace Operator version 0.9.0+, there are some changes in the CSI driver mount failure recovery:
-
The single driver container is split in two:
- A
provisioner
container, which handles download and unpacking of OneAgent binaries. - A
server
container, which handles mount and unmount requests of CSI volumes, even if theprovisioner
container runs out of memory.
- A
-
In case of issues with the CSI driver (such as network issues), the injected application pods are no longer blocked; empty files are mounted instead, so that the pods can start (there's no monitoring, but pods are available).
-
The CSI driver defaults to 10 mount attempts (about 5 minutes). Each attempt occurs when the pod restarts. For information on the exponential back-off delay, see Container restart policy.
You can configure the number of attempts in DynaKube:
apiVersion: dynatrace.com/v1beta1 kind: DynaKube metadata: name: dynakube annotations: feature.dynatrace.com/max-csi-mount-attempts: "10"
Update OneAgent if automatic updates are disabled
By default, Dynatrace Operator handles OneAgent updates automatically. If you choose to disable automatic updates, and haven't set any standard OneAgent version in Dynatrace, you can manually update OneAgent by running the command below.
kubectl -n dynatrace rollout restart daemonset/<DYNAKUBE>-oneagent
oc -n dynatrace rollout restart daemonset/<DYNAKUBE>-oneagent
Updating tokens
If you need to update your tokens, follow the steps below.
Find current tokens
Delete your secret
Create new tokens
Create a new secret with updated tokens
Delete the old tokens
Find current tokens
Find and save your currently used tokens.
Note: After generating new tokens, you'll need to delete the old ones.
kubectl -n dynatrace get secrets <dynakube-name> -o yaml | yq '.data.apiToken' | base64 -d
oc -n dynatrace get secrets <dynakube-name> -o yaml | yq '.data.apiToken' | base64 -d
Delete your secret
To delete the secret, run one of the commands below.
Note: In Kubernetes, used tokens are stored in a secret named dynakube
by default. If the DynaKube custom resource has a different name, or the tokens
field in Dynakube is set, make sure that the new secret has the same name as defined there.
kubectl -n dynatrace delete secret dynakube
oc -n dynatrace delete secret dynakube
Create new tokens
For instructions on how to create the tokens, see Tokens and permissions required.
Create a new secret with updated tokens
To create a new secret with the updated tokens, run one of the commands below, making sure to replace the placeholders with the new tokens.
- For
apiToken
:kubectl -n dynatrace create secret generic dynakube --from-literal="apiToken=<API-TOKEN>"
- For
apiToken
anddataIngestToken
:kubectl -n dynatrace create secret generic dynakube --from-literal="apiToken=<API-TOKEN>" --from-literal="dataIngestToken=<INGEST-TOKEN>"
- For
apiToken
:oc -n dynatrace create secret generic dynakube --from-literal="apiToken=<API-TOKEN>"
- For
apiToken
anddataIngestToken
:oc -n dynatrace create secret generic dynakube --from-literal="apiToken=<API-TOKEN>" --from-literal="dataIngestToken=<INGEST-TOKEN>"
Dynatrace Operator picks up the updated secrets in approximately five minutes. Removing DynaKube and reapplying it forces an instant reconciliation.
Delete the old token
After the new tokens are in place, delete the old ones.
- In Dynatrace, go to Access tokens and look for the old token.
- Select Delete.
Uninstall Dynatrace Operator
Depending on your deployment type, select one of the following.
classicFullStack
andapplicationMonitoring
without CSI drivercloudNativeFullStack
andapplicationMonitoring
with CSI driver
classicFullStack
and applicationMonitoring
without CSI driver
- Delete Dynatrace Operator.
kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml
oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml
- Run
uninstall.sh
on every node of your Kubernetes cluster where OneAgent was installed. The default location is/opt/dynatrace/oneagent/agent/uninstall.sh
.
cloudNativeFullStack
and applicationMonitoring
with CSI driver
- Remove DynaKube custom resources and clean up all remaining Dynatrace Operator–specific objects.
kubectl delete dynakube --all -n dynatrace
oc delete dynakube --all -n dynatrace
- Wait until Dynatrace Operator components have been deleted.
kubectl -n dynatrace wait pod --for=condition=delete --selector=app.kubernetes.io/name=oneagent,app.kubernetes.io/managed-by=dynatrace-operator --timeout=300s
oc -n dynatrace wait pod --for=condition=delete --selector=app.kubernetes.io/name=oneagent,app.kubernetes.io/managed-by=dynatrace-operator --timeout=300s
-
Restart your monitored applications.
-
Uninstall Dynatrace Operator and delete the Dynatrace namespace/project.
kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes-csi.yaml
kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml
kubectl delete namespace dynatrace
oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift-csi.yaml
oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml
oc delete project dynatrace
- Delete the
/var/lib/kubelet/plugins/csi.oneagent.dynatrace.com
directory from all Kubernetes nodes.
Alternative deployments
Besides the main deployment of Dynatrace Operator on Kubernetes via kubectl
and OpenShift via oc
, you have the following alternatives, depending on your needs.
- Set up Kubernetes/OpenShift monitoring with Helm
- Set up application-only monitoring (automatic/pod runtime/container build-time)
- Set up Kubernetes/OpenShift monitoring with DaemonSet
- Set up OpenShift monitoring via OperatorHub
Other configuration options
- Enable Kubernetes/OpenShift API monitoring
- Deployment options on Kubernetes/OpenShift
- DynaKube parameters for Dynatrace Operator on Kubernetes/OpenShift
- Configuration options for Dynatrace Operator on Kubernetes/OpenShift
- Migrate from OneAgent Operator to Dynatrace Operator on Kubernetes/OpenShift using kubectl/oc