How do I monitor OpenShift Container Platform?

Based on Docker and Kubernetes, Red Hat OpenShift is a next generation platform for developing, deploying, and running containerized applications, conveniently and at scale. Using the Red Hat OpenShift Container Platform (formerly OpenShift Enterprise), you can manage OpenShift on either physical or virtual infrastructure.

To monitor applications running within an OpenShift cluster, install Dynatrace OneAgent on each cluster node, as described below.

Before you begin

A Red Hat Customer Portal login is required to access the enterprise-ready dynatrace/oneagent image from the Red Hat Container Catalog (RHCC). If you do not already have a Red Hat Customer Portal login, you can register here.

Locate your Dynatrace OneAgent installer URL

If you plan to install Dynatrace OneAgent on more than 50 hosts, please consider serving the installer script via a dedicated server, such as Amazon S3. Otherwise, with more than 50 concurrent connections, Dynatrace Server may throttle requests.

The first step is to obtain the location for ONEAGENT_INSTALLER_SCRIPT_URL. This information is presented to you during Dynatrace OneAgent installation.

To get your ONEAGENT_INSTALLER_SCRIPT_URL

  1. Select Deploy Dynatrace from the navigation menu.
  2. Click Start installation and select Linux.
  3. Locate your ONEAGENT_INSTALLER_SCRIPT_URL, as shown below.

Installation

The following OpenShift Template uses a Dynatrace OneAgent Docker image with a DaemonSet to install Dynatrace OneAgent for full-stack monitoring on each node of an OpenShift cluster.

Note that enforcing the below dynatrace-oneagent.yml template requires a Service Account named dynatrace that can create privileged pods. See below for detailed instructions.

kind: Template
apiVersion: v1
name: dynatrace-oneagent
labels:
  template: dynatrace-oneagent
metadata:
  name: dynatrace-oneagent
  annotations:
    openshift.io/display-name: Dynatrace OneAgent
    description: Installs Dynatrace OneAgent for all-in-one, full-stack monitoring of OpenShift with Dynatrace. Requires privileged access.
objects:
- apiVersion: extensions/v1beta1
  kind: DaemonSet
  metadata:
    name: dynatrace-oneagent
  spec:
    template:
      metadata:
        labels:
          name: dynatrace-oneagent
      spec:
        containers:
        - name: dynatrace-oneagent
          image: registry.connect.redhat.com/dynatrace/oneagent
          imagePullPolicy: Always
          env:
          - name: ONEAGENT_INSTALLER_SCRIPT_URL
            value: "${ONEAGENT_INSTALLER_SCRIPT_URL}"
          - name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
            value: "${ONEAGENT_INSTALLER_SKIP_CERT_CHECK}"
          volumeMounts:
          - name: host-root
            mountPath: /mnt/root
          securityContext:
            privileged: true
        volumes:
        - name: host-root
          hostPath:
            path: /
        hostIPC: true
        hostNetwork: true
        hostPID: true
        serviceAccountName: dynatrace
parameters:
- name: ONEAGENT_INSTALLER_SCRIPT_URL
  description: "A URL that points to your cluster's OneAgent download location (Select \"Deploy Dynatrace\" from the Dynatrace navigation menu to access your URL). Example: https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp."
  required: true
- name: ONEAGENT_INSTALLER_SKIP_CERT_CHECK
  description: "Must be true if the SSL certificate check upon OneAgent download will be omitted, otherwise false (default). If you're using a Dynatrace Managed cluster with a self-signed certificate, set this to true."
  value: "false"
  required: false

Log into your OpenShift cluster as system:admin:

$ oc login -u system:admin

Select an OpenShift project to run the Dynatrace OneAgent image:

$ oc project openshift-infra

In this project, create a service account named dynatrace:

$ oc create serviceaccount dynatrace

Allow the dynatrace service account to pull images from the RHCC via registry.connect.redhat.com. Be sure to replace [username], [password] and [email] with your Red Hat Customer Portal's account credentials:

$ oc secrets new-dockercfg rhcc \
    --docker-server=registry.connect.redhat.com \
    --docker-username=[username] \
    --docker-password=[password] \
    --docker-email=[email]
$ oc secrets link dynatrace rhcc --for=pull

Grant the dynatrace service account permissions to run Dynatrace OneAgent as a privileged container:

$ oc adm policy add-scc-to-user privileged -z dynatrace

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml OpenShift template. Be sure to replace [oneagent-installer-script-url] with an appropriate download location:

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created

Verify that the dynatrace-oneagent daemon set has been created successfully:

$ oc status
In project openshift-infra on server https://127.0.0.1:8443

pod/dynatrace-oneagent-abcde runs dynatrace/oneagent
$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m
$ oc logs -f dynatrace-oneagent-abcde
09:46:18 Deploying agent to /tmp/Dynatrace-OneAgent-Linux.sh via https://EnvironmentID.live.dynatrace.com/installer/oneagent/unix/latest/AbCdEfGhIjKlMnOp
...
09:46:24 Validating agent installer in /tmp/Dynatrace-OneAgent-Linux.sh
Verification successful
09:46:24 Started agent deployment as docker image, PID 1234.
09:46:24 Container version: 1.x
09:46:24 Checking root privileges...
09:46:24 OK
09:46:27 Installation started, version 1.x, build date: 01.01.2017, PID 1234.
...

Uninstallation

Uninstalling Dynatrace OneAgent from each node of an OpenShift cluster can be achieved as follows:

Select the project that runs the dynatrace-oneagent daemon set:

$ oc project openshift-infra

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Updating

Whenever a new version of Dynatrace OneAgent becomes available in Dynatrace, you can re-deploy Dynatrace OneAgent as explained in the steps below. Your dynatrace/oneagent image will automatically fetch the latest version of Dynatrace OneAgent. If you've specified a default OneAgent install version for new hosts and applications in your OneAgent updates settings, the dynatrace/oneagent image will automatically fetch the defined default version of Dynatrace OneAgent:

Delete the dynatrace-oneagent daemon set:

$ oc delete ds/dynatrace-oneagent

Deploy Dynatrace OneAgent using the above dynatrace-oneagent.yml OpenShift template. Be sure to replace [oneagent-installer-script-url] with an appropriate download location:

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created

Pitfalls

Find out how to solve common problems that you may encounter.

Deployment seems successful, the dynatrace/oneagent image cannot be pulled

$ oc get pods
NAME                       READY   STATUS         RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ErrImagePull   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: image can't be pulled

This is typically the case if the dynatrace service account has not been allowed to pull images from the RHCC (please see the installation steps above):

Deployment seems successful, but the dynatrace-oneagent container does not produce meaningful logs

$ oc get pods
NAME                       READY   STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   0/1     ContainerCreating   0          3s
$ oc logs -f dynatrace-oneagent-abcde
Error from server (BadRequest): container "dynatrace-oneagent" in pod "dynatrace-oneagent-abcde" is waiting to start: ContainerCreating

This is typically the case if the container has not yet fully started. Simply wait a couple of seconds.

Deployment seems successful, but the dynatrace-oneagent container is not running

$ oc process -f dynatrace-oneagent.yml -p=ONEAGENT_INSTALLER_SCRIPT_URL=[oneagent-installer-script-url] | oc create -f -
daemonset "dynatrace-oneagent" created
$ oc get pods
No resources found.

This is typically the case if the dynatrace service account has not been configured to run privileged pods (please see the installation steps above):

$ oc describe ds/dynatrace-oneagent
Name:   dynatrace-oneagent
Image(s): dynatrace/oneagent
Selector: name=dynatrace-oneagent
Node-Selector:  <none>
Labels:   template=dynatrace-oneagent
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Events:
  FirstSeen LastSeen  Count From    SubObjectPath Type    Reason    Message
  --------- --------  ----- ----    ------------- --------  ------    -------
  6m    3m    17  {daemon-set }     Warning   FailedCreate  Error creating: pods "dynatrace-oneagent-" is forbidden: unable to validate against any security context constraint: [spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.hostIPC: Invalid value: true: Host IPC is not allowed to be used]

Deployment was successful, but monitoring data is not available in Dynatrace

$ oc get pods
NAME                       READY     STATUS              RESTARTS   AGE
dynatrace-oneagent-abcde   1/1       Running             0          1m

This is typically caused by a timing issue that occurs if application containers have started before Dynatrace OneAgent was fully installed on the system. As a consequence, some parts of your application run uninstrumented. To be on the safe side, Dynatrace OneAgent should be fully integrated before you start your application containers. If your application has already been running, restarting its containers will have the very same effect.