Deploy OneAgent on Kubernetes for application-only monitoring

Dynatrace supports Full-Stack Monitoring for Kubernetes, from the application down to the infrastructure layer. However, if you don't have access to the infrastructure layer, Dynatrace also provides the option of application-only monitoring. See below for instructions on how to set up Dynatrace to monitor your applications running on Kubernetes.

Note: When deployed in application-only mode, OneAgent monitors the memory, disk, CPU, and networking of processes within the container only. Host metrics aren't monitored.

Prerequisites

Note: This deployment strategy requires extra ephemeral storage:

  • ~325 MB for glibc
  • ~290 MB for musl
  • ~650 MB for glibc and musl combined

Integrate OneAgent into your application

The following options explain how you can integrate OneAgent with Kubernetes applications.

The application-only injection strategy uses OneAgent Operator. Support for using Dynatrace Operator to handle application-only injection is in development.

Automatic application-only injection

Dynatrace offers the option to inject OneAgent into Kubernetes pods. OneAgent Operator runs an admission controller that can modify pods to inject OneAgent by adding an init container. This init container will download the OneAgent package and configure the other containers to be monitored.

Required versions
  • Kubernetes version 1.14+
  • OneAgent Operator v0.8.0+
  1. Create a Dynatrace namespace.
kubectl create namespace dynatrace
  1. Install OneAgent Operator.

OneAgent Operator acts on its dynatrace namespace. You can also observe the logs of OneAgent Operator.

kubectl apply -f https://github.com/Dynatrace/dynatrace-oneagent-operator/releases/latest/download/kubernetes.yaml
kubectl -n dynatrace logs -f deployment/dynatrace-oneagent-operator
  1. Create the secret holding the PaaS token for authentication to the Dynatrace Cluster.
kubectl -n dynatrace create secret generic oneagent --from-literal="paasToken=PAAS_TOKEN"
  1. Save the OneAgent custom resource definition. The rollout of Dynatrace OneAgent for application-only installations is governed by the custom resource type OneAgentAPM. Retrieve the cr-apm.yaml file from the GitHub repository.
curl -o cr-apm.yaml https://raw.githubusercontent.com/Dynatrace/dynatrace-oneagent-operator/master/deploy/cr-apm.yaml
  1. Adapt the values of the custom resource as follows:
  • required Specify the spec.apiUrl parameter, which is the URL of your Dynatrace environment, for your SaaS, Managed, or ActiveGate instance.

  • optional Configure spec.useImmutableImage to true to pull a OneAgent Docker image from your Dynatrace environment. Use this parameter together with the agentVersion parameter to control the version of OneAgent.

  • optional Configure spec.agentVersion using semantic versioning (major.minor.patch - example: 1.203.0). If no version is specified, the OneAgent defaults to the latest version available.

  • optional Configure network zones by setting the spec.networkZone parameter to your network zone.

spec:
  networkZone: <your_network_zone>

See network zones for more information.

  1. Label your namespaces.

OneAgent Operator injects into all pods that belong to namespaces labeled oneagent.dynatrace.com/instance. The value for this label must be the name of the OneAgentAPM instance that you want to use to configure the corresponding namespaces. You must label all namespaces you want to monitor. Note that the namespaces can point to different OneAgentAPM instances.

kubectl label namespace default oneagent.dynatrace.com/instance=oneagentapm
  1. Configure the injection.

You can configure the injection through Kubernetes annotations.

Note: These settings apply to all containers running on the corresponding pods.

  • oneagent.dynatrace.com/inject: <"true"> or <"false">. Sets the default injection for pods on the namespace. Can be overridden by adding the annotation to the pods themselves. It defaults to true.

Example

kubectl annotate namespace default oneagent.dynatrace.com/inject=false
namespace/default annotated
  1. Create the custom resource
kubectl apply -f cr-apm.yaml
  1. Deploy your applications.

All deployed pods will then be monitored.

  1. For troubleshooting purposes, you can view OneAgent logs, which by default are on /opt/dynatrace/oneagent-paas/log inside the instrumented containers.

Pod runtime injection

Required versions

OneAgent version 1.149+

With the container runtime integration, OneAgent is made available to the application container via an initContainer—your application image remains unaffected.

To integrate OneAgent into your application at runtime, extend your deployment template as follows.

# your application containers
      containers:
      - name: customer-app
        image: tomcat
        env:
        - name: LD_PRELOAD
          value: /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
        - name: DT_NETWORK_ZONE
          value: <your_network_zone>
        volumeMounts:
        - mountPath: /opt/dynatrace/oneagent
          name: oneagent

# initcontainer to download OneAgent
      initContainers:
      - name: install-oneagent
        image: alpine:latest
        command:
        - /bin/sh
        args:
        - -c
        - ARCHIVE=$(mktemp) && wget -O $ARCHIVE "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_PAAS_TOKEN&$DT_ONEAGENT_OPTIONS" && unzip -o -d /opt/dynatrace/oneagent $ARCHIVE && rm -f $ARCHIVE
        env:
        - name: DT_API_URL
          value: https://<Your-environment-ID>.live.dynatrace.com/api
        - name: DT_PAAS_TOKEN
          value: <paastoken>
        - name: DT_ONEAGENT_OPTIONS
          value: flavor=<FLAVOR>&include=<TECHNOLOGY>
        volumeMounts:
        - mountPath: /opt/dynatrace/oneagent
          name: oneagent

# Make OneAgent available as a volume
      volumes:
      - name: oneagent
        emptyDir: {}
  • In the # initContainer to download OneAgent and # Make OneAgent available as a volume sections, add the initContainer, which will download OneAgent and make it available as a volume.

  • In the DT_ONEAGENT_OPTIONS section, set the OneAgent code module required for your compiler flavor (FLAVOR) and application (TECHNOLOGY).

    • Valid options for flavor are default, musl, or multidistro. Set default to download glibc binaries or set musl to download musl binaries. Set multidistro to download both the musl and glibc binaries and subsequently autodetect which binaries to use. Note that image size will be larger in this case, as it includes both flavors.
    • Valid options for technology are all, java, apache, nginx, nodejs, dotnet, php, go, and sdk.
    • For ARM, use the following value: flavor=<FLAVOR>&include=<TECHNOLOGY>&arch=arm. For other architectures, see the list of valid values (scroll down to the arch parameter).
    • If you want to specify several code modules, use the following syntax: &include=technology1&include=technology2.

Note: If you include specific technology-support options rather than 'support for all technologies' options, you'll get a smaller OneAgent package.

What if my Docker image is based on Alpine Linux?
Dynatrace OneAgent supports the flavor musl for Alpine Linux based environments.
Valid options for technology are all, dotnet, go, php, java, apache, nginx, and nodejs.

  • In the # your application containers section, add the newly created volume to the container of your application. Also add the LD_PRELOAD environment variable.

  • optional In the # your application containers section, configure network zones:

containers:
  env:
  - name: DT_NETWORK_ZONE
    value: <your_network_zone>

See network zones for more information.

  • optional Configure a proxy address.

In case you run an environment with proxy, you need to set the DT_PROXY environment variable in the application container to pass the proxy credentials to OneAgent.

Note: For Alpine Linux-based containers, you might need to update the wget shipped with the Alpine image to allow for proxy authentication for the download of OneAgent.

Container build-time injection

Required versions
  • Docker version 17.05+
  • OneAgent version 1.155+

Follow the steps below to integrate OneAgent into the application image.

  1. Sign in to Docker with your Dynatrace environment ID as username and PaaS token as password.

    docker login -u <environmentID> <ADDRESS>
    
  2. Add the following lines of code to the application image, after the last FROM command:

    COPY --from=<ADDRESS>/linux/oneagent-codemodules:<TECHNOLOGY> / /
    ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
    

    where:

  • <ADDRESS> is:
    • EnvironmentActiveGate: <ActiveGateaddress:9999>
    • SAAS: {yourenvid}.live.dynatrace.com
    • Managed: {ManagedAddress}
  • <TECHNOLOGY> is: The OneAgent code module required for your application. Valid options are all, java, apache, nginx, nodejs, dotnet, php, go, and sdk. You can specify several code modules, separated by hyphen (-), for example java-go. Including specific technology-support options, rather than support for all technology options, results in a smaller OneAgent package.

What if my Docker image is based on Alpine Linux?
Dynatrace OneAgent supports Alpine Linux based environments.
Use this syntax:

COPY --from=<ACTIVEGATE-ADDRESS>/linux/oneagent-codemodules-musl:<TECHNOLOGY> / /
ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so

Valid options here are all, dotnet, php, java, apache, nginx, nodejs, and go.

  1. Build your application image.

Build the Docker image from your Dockerfile to use it in your Kubernetes environment.

docker build -t yourapp .

You can monitor your application containers with a different Dynatrace environment.

For OneAgent version 1.139 or higher, if you have an existing application image where you have already added the OneAgent code modules for a specific Dynatrace environment, you can have the OneAgent report to another Dynatrace environment without rebuilding your application image.
For this, you need to make a call to the REST endpoint of your second Dynatrace environment. Don't forget to adapt the respective placeholders <environmentID> and <token>.

curl "https://<environmentID>.live.dynatrace.com/api/v1/deployment/installer/agent/connectioninfo?Api-Token=<token>"

In return, you get a JSON object that covers the required information that needs to be passed as an environment variable to the application container.
Make sure you set the environment variables of the application container as described below:

  • DT_TENANT: equals tenantUUID
  • DT_TENANTTOKEN: equals tenantToken
  • DT_CONNECTION_POINT: semi-colon separated list of communicationEndpoints
  1. optional Configure network zones

You can configure network zones as an environment variable:

  • DT_NETWORK_ZONE: equals your.network.zone

See network zones for more information.

  1. optional Configure a proxy address

In case you run an environment with proxy, you need to set the DT_PROXY environment variable in the application container to pass the proxy credentials to OneAgent.

Note: For Alpine Linux-based containers, you might need to update the wget shipped with the Alpine image to allow for proxy authentication for the download of OneAgent.

Update OneAgent

Automated application-only and run-time injections

Each time you want to leverage a new OneAgent version, you only need to redeploy your pods. In both automated application-only and run-time injections, OneAgent is downloaded and injected within the definition of an initContainer. By default, the latest version of OneAgent is downloaded, but you can control which OneAgent version is downloaded by specifying it either in the spec.agentVersion parameter of the custom resource or in the download URL, depending on the injection method you used.

Container build-time injection

You need to manually update OneAgent by rebuilding the container image every time a new version of a code module is needed.

Uninstall OneAgent

To uninstall OneAgent from application-only monitoring, simply remove references from your application or Docker image and redeploy the application.

Container build-time injection

  1. Remove the two lines of code from the application image.
COPY --from=<ACTIVEGATE-ADDRESS>/linux/oneagent-codemodules:<TECHNOLOGY> / /
ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
  1. Rebuild the application image.
docker build -t yourapp .

Pod run-time injection

Remove the install-oneagent YAML from your deployment template.

k8s-app-only-run-time.yaml
Download
# your application containers
      containers:
      - name: customer-app
        image: tomcat
        env:
        - name: LD_PRELOAD
          value: /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
        volumeMounts:
        - mountPath: /opt/dynatrace/oneagent
          name: oneagent

# initContainer to download OneAgent
      initContainers:
      - name: install-oneagent
        image: alpine:3.8
        command:
        - /bin/sh
        args:
        - -c
        - ARCHIVE=$(mktemp) && wget -O $ARCHIVE "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_PAAS_TOKEN&$DT_ONEAGENT_OPTIONS" && unzip -o -d /opt/dynatrace/oneagent $ARCHIVE && rm -f $ARCHIVE
        env:
        - name: DT_API_URL
          value: https://<Your-environment-ID>.live.dynatrace.com/api
        - name: DT_PAAS_TOKEN
          value: <paastoken>
        - name: DT_ONEAGENT_OPTIONS
          value: flavor=<FLAVOR>&include=<TECHNOLOGY>
        volumeMounts:
        - mountPath: /opt/dynatrace/oneagent
          name: oneagent

# Make OneAgent available as a volume
      volumes:
      - name: oneagent
        emptyDir: {}

Automated application-only injection

  1. Uninstall OneAgent Operator.
kubectl delete -f https://github.com/Dynatrace/dynatrace-oneagent-operator/releases/latest/download/kubernetes.yaml
  1. Delete the dynatrace namespace. This will also delete OneAgentAPM objects.
  2. Optional Remove the corresponding labels and annotations from namespaces/pods.
  3. Redeploy your pods.