How do I deploy Dynatrace OneAgent on OpenShift Container Platform?

Based on Docker and Kubernetes, Red Hat OpenShift is a next generation platform for developing, deploying, and running containerized applications, conveniently and at scale. Using the Red Hat OpenShift Container Platform (formerly OpenShift Enterprise), you can manage OpenShift on either physical or virtual infrastructure.

To monitor applications running within an OpenShift cluster, install Dynatrace OneAgent on each cluster node, as described below.

Before you begin

A Red Hat Customer Portal login is required to access the enterprise-ready dynatrace/oneagent image from the Red Hat Container Catalog (RHCC). If you don't already have a Red Hat Customer Portal login, you can register here.

Locate your Dynatrace OneAgent installer URL

The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you during Dynatrace OneAgent installation.

To get your ONEAGENT_INSTALLER_SCRIPT_URL

  1. Select Deploy Dynatrace from the navigation menu.
  2. Click Start installation and select Linux.
  3. Copy the URL, shown below. This is your ONEAGENT_INSTALLER_SCRIPT_URL.

Installation

The following OpenShift Template uses a Dynatrace OneAgent Docker image with a DaemonSet to install Dynatrace OneAgent for full-stack monitoring on each node of an OpenShift cluster.

Note that enforcing the below dynatrace-oneagent.yml template requires a Service Account named dynatrace that can create privileged pods. See below for detailed instructions.

Log into your OpenShift cluster as system:admin:

$ oc login -u system:admin

Select an OpenShift project to run the Dynatrace OneAgent image:

$ oc project openshift-infra

In this project, create a service account named dynatrace:

$ oc create serviceaccount dynatrace

Allow the dynatrace service account to pull images from the RHCC via registry.connect.redhat.com. Be sure to replace [username], [password] and [email] with your Red Hat Customer Portal's account credentials:

$ oc secrets new-dockercfg rhcc \
    --docker-server=registry.connect.redhat.com \
    --docker-username=[username] \
    --docker-password=[password] \
    --docker-email=[email]
$ oc secrets link dynatrace rhcc --for=pull

Grant the dynatrace service account permissions to run Dynatrace OneAgent as a privileged container:

$ oc adm policy add-scc-to-user privileged -z dynatrace

Deploy Dynatrace OneAgent using the dynatrace-oneagent.yml OpenShift template. Replace [oneagent-installer-script-url] with your actual OneAgent installer script URL. Be sure to enclose the URL with quotation marks. Otherwise the URL will break the commands.

$ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -

Verify that the dynatrace-oneagent daemon set has been created successfully:

$ oc status
In project openshift-infra on server https://127.0.0.1:8443

pod/dynatrace-oneagent-abcde runs dynatrace/oneagent
$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m
$ oc logs -f dynatrace-oneagent-abcde
09:46:18 Deploying agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp
...
09:46:24 Validating agent installer in /tmp/Dynatrace-OneAgent-Linux.sh
Verification successful
09:46:24 Started agent deployment as docker image, PID 1234.
09:46:24 Container version: 1.x
09:46:24 Checking root privileges...
09:46:24 OK
09:46:27 Installation started, version 1.x, build date: 01.01.2017, PID 1234.
...

Note: For OpenShift versions 3.7 and higher, the Red Hat Connect Partner Registry may not accept the auto-generated dockercfg secret type (because of BZ#1476330). Therefore, you must create a generic file-based secret using the generated file from a docker login command.

Uninstallation

Uninstalling Dynatrace OneAgent from each node of an OpenShift cluster can be achieved as follows:

Select the project that runs the dynatrace-oneagent daemon set:

$ oc project openshift-infra

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Updating

Whenever a new version of Dynatrace OneAgent becomes available in Dynatrace, you can re-deploy Dynatrace OneAgent as explained in the steps below. Your dynatrace/oneagent image will automatically fetch the latest version of Dynatrace OneAgent. If you've specified a default OneAgent install version for new hosts and applications in your OneAgent updates settings, the dynatrace/oneagent image will automatically fetch the defined default version of Dynatrace OneAgent:

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml OpenShift template. Be sure to replace [oneagent-installer-script-url] with an appropriate download location:

$ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -
daemonset "dynatrace-oneagent" created

Please note that quotes are needed to protect special shell characters within the Dynatrace OneAgent installer URL.

Troubleshooting

Find out how to solve problems that you may encounter when deploying OneAgent on an OpenShift Container Platform.

Deployment seems successful, however the dynatrace/oneagent image can't be pulled

$ oc get pods
NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account hasn't been allowed to pull images from the RHCC (please see the installation steps above):

Deployment seems successful, but the dynatrace-oneagent container doesn't produce meaningful logs

$ oc get pods
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container hasn't yet fully started. Simply wait a few more seconds.

Deployment seems successful, but the dynatrace-oneagent container isn't running

$ oc process -f dynatrace-oneagent.yml ONEAGENT_INSTALLER_SCRIPT_URL="[oneagent-installer-script-url]" | oc create -f -
daemonset "dynatrace-oneagent" created

Please note that quotes are needed to protect the special shell characters in the Dynatrace OneAgent installer URL.

$ oc get pods
No resources found.

This is typically the case if the dynatrace service account hasn't been configured to run privileged pods (please see the installation steps above):

$ oc describe ds/dynatrace-oneagent
Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
  FirstSeen LastSeen  Count From    SubObjectPath Type    Reason    Message
  --------- --------  ----- ----    ------------- --------  ------    -------
  6m    3m    17  {daemon-set }     Warning   FailedCreate  Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]

Deployment was successful, but monitoring data isn't available in Dynatrace

$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs when application containers are started before Dynatrace OneAgent is fully installed on the system. As a consequence, some parts of your application may be uninstrumented. To be on the safe side, Dynatrace OneAgent should be fully installed and configured before you start your application containers. If your application is already running, restart its containers to achieve the same outcome.

Note:
If you plan to install Dynatrace OneAgent on more than 50 hosts, please consider serving the installer script via a dedicated server, such as Amazon S3. Otherwise, with more than 50 concurrent connections, Dynatrace Server may throttle requests.

Limitations

The same limitations apply as when deploying OneAgent as a Docker container.