Set up application-only monitoring on Kubernetes/OpenShift
Dynatrace supports Full-Stack Monitoring for Kubernetes, from the application down to the infrastructure layer. However, if you don't have access to the infrastructure layer, Dynatrace also provides the option of application-only monitoring. See below for instructions on how to set up Dynatrace to monitor your applications running on Kubernetes.
Note: When deployed in application-only mode, OneAgent monitors the memory, disk, CPU, and networking of processes within the container only. Host metrics aren't monitored.
To integrate OneAgent with Kubernetes applications you have the following options:
-
Container build-time injection
Note: For PPCLE and S390x architectures, pod runtime and container build-time injections are supported.
Automatic application-only injection
This section describes how to deploy OneAgent for application-only monitoring by automatic injection using Dynatrace Operator.
Dynatrace offers the option to inject OneAgent into Kubernetes pods using Kubernetes-native admission controllers. Dynatrace Operator runs a mutating webhook that can modify pods to inject OneAgent by adding an init container. This init container will download the OneAgent package and configure the other containers to be monitored.
Prerequisites
- Kubernetes version 1.20+ or OpenShift version 4.7+
- Review the list of supported applications and versions.
- Review the storage requirements.
- Review the tokens and permissions required.
Deploy
Automatic application-only monitoring is activated using the Dynatrace Operator applicationMonitoring
mode. Instructions for this mode, together with all other modes, are part of Set up Kubernetes monitoring.
- At step 5 of the instructions, make sure you download and configure the
applicationMonitoring
Dynakube custom resource. - To start monitoring, you need to configure monitoring for namespaces and pods.
Limitations
Dynatrace Operator doesn't support OpenShift version 3.11 or earlier.
Update
Each time you want to leverage a new OneAgent version, you only need to redeploy your pods. With automated application monitoring, OneAgent is downloaded and injected within an init container. By default, the latest version of OneAgent is downloaded, but you can control which OneAgent version is downloaded by specifying it with the version
parameter of the custom resource.
If you're using CSI drivers, the new OneAgent is downloaded automatically, but you need to recycle the pods to use the latest version.
Uninstall
To uninstall OneAgent from application-only monitoring, simply remove references from your application or Docker image and redeploy the application.
-
To uninstall Dynatrace Operator in deployments where a CSI driver isn't used, run one of the following commands:
kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml
oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml
-
To uninstall Dynatrace Operator in deployments where a CSI driver is used
-
Remove DynaKube custom resources and clean up all remaining Dynatrace Operator–specific objects.
kubectl delete dynakube --all -n dynatrace
oc delete dynakube --all -n dynatrace
-
Restart your monitored applications.
-
Uninstall Dynatrace Operator and delete the Dynatrace namespace/project.
kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes-csi.yaml kubectl delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/kubernetes.yaml kubectl delete namespace dynatrace
oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift-csi.yaml oc delete -f https://github.com/Dynatrace/dynatrace-operator/releases/download/v0.11.2/openshift.yaml oc delete project dynatrace
-
Pod runtime injection
Inject OneAgent code modules into a container as it is deployed.
Note: This method of instrumenting applications has limitations in linking Kubernetes workloads with monitored containers/processes. To gain proper relationships and linking, use the automatic application-only injection.
Prerequisites
-
Review the list of supported applications and versions.
-
Storage requirements:
- ~325 MB for glibc
- ~290 MB for musl
- ~650 MB for glibc and musl combined
Deploy
To integrate OneAgent into your application at runtime, select one of the options below, depending on your platform.
OneAgent is made available to the application container via an initContainer
—your application image remains unaffected.
To integrate OneAgent into your application at runtime, extend your deployment template as follows.
# your application containers
containers:
- name: customer-app
image: tomcat
env:
- name: LD_PRELOAD
value: /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
- name: DT_NETWORK_ZONE
value: <your_network_zone>
volumeMounts:
- mountPath: /opt/dynatrace/oneagent
name: oneagent
# initcontainer to download OneAgent
initContainers:
- name: install-oneagent
image: alpine:latest
command:
- /bin/sh
args:
- -c
- ARCHIVE=$(mktemp) && wget -O $ARCHIVE "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_PAAS_TOKEN&$DT_ONEAGENT_OPTIONS" && unzip -o -d /opt/dynatrace/oneagent $ARCHIVE && rm -f $ARCHIVE
env:
- name: DT_API_URL
value: https://<Your-environment-ID>.live.dynatrace.com/api
- name: DT_PAAS_TOKEN
value: <paastoken>
- name: DT_ONEAGENT_OPTIONS
value: flavor=<FLAVOR>&include=<TECHNOLOGY>
volumeMounts:
- mountPath: /opt/dynatrace/oneagent
name: oneagent
# Make OneAgent available as a volume
volumes:
- name: oneagent
emptyDir: {}
-
In the
# initContainer to download OneAgent
and# Make OneAgent available as a volume
sections, add theinitContainer
, which will download OneAgent and make it available as a volume. -
In the
DT_ONEAGENT_OPTIONS
section, set the OneAgent code module required for your compiler flavor (FLAVOR
) and application (TECHNOLOGY
).- Valid options for
flavor
aredefault
,musl
, ormultidistro
. Setdefault
to downloadglibc
binaries or setmusl
to downloadmusl
binaries. Setmultidistro
to download both themusl
andglibc
binaries and subsequently autodetect which binaries to use. Note that image size will be larger in this case, as it includes both flavors. - Valid options for
technology
areall
,java
,apache
,nginx
,nodejs
,dotnet
,php
,go
, andsdk
. - For ARM, use the following value:
flavor=default&arch=arm&include=<TECHNOLOGY>
. For other architectures, see the list of valid values (scroll down to thearch
parameter). - If you want to specify several code modules, use the following syntax:
&include=technology1&include=technology2
.
- Valid options for
Note: If you include specific technology-support options rather than 'support for all technologies' options, you'll get a smaller OneAgent package.
What if my Docker image is based on Alpine Linux?
Dynatrace OneAgent supports the flavor musl
for Alpine Linux based environments.
Valid options for technology
are all
, dotnet
, go
, php
, java
, apache
, nginx
, and nodejs
.
-
In the
# your application containers
section, add the newly created volume to the container of your application. Also add theLD_PRELOAD
environment variable. -
optional In the
# your application containers
section, configure network zones:
containers:
env:
- name: DT_NETWORK_ZONE
value: <your_network_zone>
See network zones for more information.
- optional Configure a proxy address.
In case you run an environment with proxy, you need to set the DT_PROXY
environment variable in the application container to pass the proxy credentials to OneAgent.
Note: For Alpine Linux-based containers, you might need to update the wget
shipped with the Alpine image to allow for proxy authentication for the download of OneAgent.
Extend your deployment template as follows.
Note: This option refers to .NET applications in Windows containers.
# your application containers
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample
labels:
app: sample
spec:
replicas: 1
template:
metadata:
name: sample
labels:
app: sample
spec:
nodeSelector:
"kubernetes.io/os": windows
containers:
- name: sample
image: mcr.microsoft.com/dotnet/framework/samples:aspnetapp
env:
#.NET Framework
- name: COR_ENABLE_PROFILING
value: "0x01"
- name: COR_PROFILER
value: "{B7038F67-52FC-4DA2-AB02-969B3C1EDA03}"
- name: COR_PROFILER_PATH_32
value: "C:\\oneagent\\agent\\lib\\oneagentloader.dll"
- name: COR_PROFILER_PATH_64
value: "C:\\oneagent\\agent\\lib64\\oneagentloader.dll"
#.NET Core
- name: CORECLR_ENABLE_PROFILING
value: "0x01"
- name: CORECLR_PROFILER
value: "{B7038F67-52FC-4DA2-AB02-969B3C1EDA03}"
- name: CORECLR_PROFILER_PATH_32
value: "C:\\oneagent\\agent\\lib\\oneagentloader.dll"
- name: CORECLR_PROFILER_PATH_64
value: "C:\\oneagent\\agent\\lib64\\oneagentloader.dll"
- name: DT_AGENTACTIVE
value: "true"
- name: DT_BLOCKLIST
value: "powershell*"
volumeMounts:
- mountPath: "C:\\OneAgent"
name: oneagent
# initcontainer to download OneAgent
initContainers:
- name: install-oneagent
image: mcr.microsoft.com/windows/servercore:ltsc2019
command:
- powershell
args:
- |
Write-Host "Trustng all certificates..."
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
Write-Host "Downloading agent..."
Invoke-WebRequest -Uri "$Env:DT_API_URL/v1/deployment/installer/agent/windows/paas/latest?Api-Token=$Env:DT_PAAS_TOKEN&$Env:DT_ONEAGENT_OPTIONS" -OutFile "installer.zip"
Write-Host "Unpacking agent..."
Expand-Archive -Path "installer.zip" -DestinationPath "C:\OneAgent" -Force
Write-Host "Configuring agent..."
$manifest = Get-Content "C:\OneAgent\manifest.json" | ConvertFrom-Json
$config = New-Item -Path "C:\OneAgent\agent\conf" -Name "standalone.conf" -Force
Add-Content -Path $config -Value "tenant $($manifest.tenantUUID)"
Add-Content -Path $config -Value "tenanttoken $($manifest.tenantToken)"
Add-Content -Path $config -Value "server $($manifest.communicationEndpoints -Join ';')"
Add-Content -Path $config -Value "storage C:\OneAgent"
Add-Content -Path $config -Value "loglevelcon NONE"
env:
- name: DT_API_URL
value: https://<YourDTTenant>.live.dynatrace.com/api
- name: DT_PAAS_TOKEN
value: <PaaSToken (Dynatrace -> Settings -> Integration -> Platform as a Service Token)
- name: DT_ONEAGENT_OPTIONS
value: flavor=x86&include=dotnet
volumeMounts:
- mountPath: "C:\\OneAgent"
name: oneagent
# Make OneAgent available as a volume
volumes:
- name: oneagent
emptyDir: {}
selector:
matchLabels:
app: sample
---
apiVersion: v1
kind: Service
metadata:
name: sample
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 80
selector:
app: sample
- In the
# initContainer to download OneAgent
and# Make OneAgent available as a volume
sections, add theinitContainer
, which will download OneAgent and make it available as a volume. - The
# your application containers
section contains environment variables that enable monitoring of .NET Framework and .NET Core applications. They can be set at the same time. For .NET Core, theCOR_ prefix
changes toCORECLR_
, for exampleCORECLR_ENABLE_PROFILING
.
Note: To report the correct memory limits in Kubernetes
-
You have to specify the limit in the deployment.
Example:
spec: containers: ... resources: limits: memory: "32Gi" requests: memory: "4Gi"
-
You have to enable access to the Kubernetes API so that OneAgent can read that value.
Update
Each time you want to leverage a new OneAgent version, you only need to redeploy your pods. In runtime injections, OneAgent is downloaded and injected within an initContainer. By default, the latest version of OneAgent is downloaded, but you can control which OneAgent version is downloaded by specifying it in the download URL.
Uninstall
To uninstall OneAgent from application-only monitoring
- Remove the install-oneagent YAML from your deployment template.
# your application containers
containers:
- name: customer-app
image: tomcat
env:
- name: LD_PRELOAD
value: /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
volumeMounts:
- mountPath: /opt/dynatrace/oneagent
name: oneagent
# initContainer to download OneAgent
initContainers:
- name: install-oneagent
image: alpine:3.8
command:
- /bin/sh
args:
- -c
- ARCHIVE=$(mktemp) && wget -O $ARCHIVE "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_PAAS_TOKEN&$DT_ONEAGENT_OPTIONS" && unzip -o -d /opt/dynatrace/oneagent $ARCHIVE && rm -f $ARCHIVE
env:
- name: DT_API_URL
value: https://<Your-environment-ID>.live.dynatrace.com/api
- name: DT_PAAS_TOKEN
value: <paastoken>
- name: DT_ONEAGENT_OPTIONS
value: flavor=<FLAVOR>&include=<TECHNOLOGY>
volumeMounts:
- mountPath: /opt/dynatrace/oneagent
name: oneagent
# Make OneAgent available as a volume
volumes:
- name: oneagent
emptyDir: {}
- Redeploy your application.
Container build-time injection
Inject OneAgent code modules into a container as it builds.
Note: This method of instrumenting applications has limitations in linking Kubernetes workloads with monitored containers/processes. To gain proper relationships and linking, use automatic application-only injection.
Prerequisites
-
Docker version 17.05+
-
Review the list of supported applications and versions.
-
Storage requirements:
- ~325 MB for glibc
- ~290 MB for musl
- ~650 MB for glibc and musl combined
-
For ARM architecture, make sure you have
wget
andunzip
installed.
Deploy
To integrate OneAgent into the application image, follow the steps below.
-
Sign in to Docker with your Dynatrace environment ID as username and PaaS token as password.
docker login -u <environmentID> <ADDRESS>
-
Add the following lines of code to the application image, after the last
FROM
command:COPY --from=<ADDRESS>/linux/oneagent-codemodules:<TECHNOLOGY> / / ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
where:
-
<ADDRESS>
is:- EnvironmentActiveGate:
<ActiveGateaddress:9999>
- SAAS:
{yourenvid}.live.dynatrace.com
- Managed:
{ManagedAddress}
- EnvironmentActiveGate:
-
<TECHNOLOGY>
is: The OneAgent code module required for your application. Valid options areall
,java
,apache
,nginx
,nodejs
,dotnet
,php
,go
, andsdk
. You can specify several code modules, separated by hyphen (-
), for examplejava-go
. Including specific technology-support options, rather than support for all technology options, results in a smaller OneAgent package.What if my Docker image is based on Alpine Linux?
Dynatrace OneAgent supports Alpine Linux based environments.
Use this syntax:COPY --from=<ACTIVEGATE-ADDRESS>/linux/oneagent-codemodules-musl:<TECHNOLOGY> / / ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
Valid options here are
all
,dotnet
,php
,java
,apache
,nginx
,nodejs
, andgo
.
-
Build your application image.
Build the Docker image from your Dockerfile to use it in your Kubernetes environment.
docker build -t yourapp .
You can monitor your application containers with a different Dynatrace environment.
For OneAgent version 1.139 or higher, if you have an existing application image where you have already added the OneAgent code modules for a specific Dynatrace environment, you can have the OneAgent report to another Dynatrace environment without rebuilding your application image.
For this, you need to make a call to the REST endpoint of your second Dynatrace environment. Don't forget to adapt the respective placeholders<environmentID>
and<token>
.curl "https://<environmentID>.live.dynatrace.com/api/v1/deployment/installer/agent/connectioninfo?Api-Token=<token>"
In return, you get a JSON object that covers the required information that needs to be passed as an environment variable to the application container.
Make sure you set the environment variables of the application container as described below:DT_TENANT
: equalstenantUUID
DT_TENANTTOKEN
: equalstenantToken
DT_CONNECTION_POINT
: semi-colon separated list ofcommunicationEndpoints
- optional Configure network zones
You can configure network zones as an environment variable:
DT_NETWORK_ZONE
: equalsyour.network.zone
See network zones for more information.
-
optional Configure a proxy address
In case you run an environment with proxy, you need to set the
DT_PROXY
environment variable in the application container to pass the proxy credentials to OneAgent.Note: For Alpine Linux-based containers, you might need to update the
wget
shipped with the Alpine image to allow for proxy authentication for the download of OneAgent.
-
Define variables with optional default values using
ARG
instructions, as shown below.ARG DT_API_URL="https://<environmentID>.live.dynatrace.com/api" ARG DT_PAAS_TOKEN="<token>" ARG DT_ONEAGENT_OPTIONS="flavor=default&include=<technology1>&include=<technology2>" ENV DT_HOME="/opt/dynatrace/oneagent"
- You can override the default values within the OpenShift
BuildConfig
. Replace<environmentID>
with your Dynatrace environment ID. If you’re using Dynatrace Managed, you need to provide your Dynatrace Server URL (https://<YourDynatraceServerURL>/e/<environmentID>/api
). Replace<token>
with the PaaS token mentioned above. - Technology support is enabled via
include
parameters. Valid options forflavor=default
areall
,java
,apache
,nginx
,nodejs
,dotnet
,php
,go
, andsdk
. Including specific technology-support options, rather than support for all technology options, results in a smaller OneAgent package.
What if my Docker image is based on Alpine Linux?
OneAgent supports the flavormusl
for Alpine Linux based environments. Valid options forflavor=musl
areall
,java
,apache
,nginx
, andnodejs
. - You can override the default values within the OpenShift
-
Add the following commands to your current Dockerfile to integrate OneAgent and activate instrumentation of your application.
ARG DT_API_URL="https://<environmentID>.live.dynatrace.com/api" ARG DT_API_TOKEN="<token>" ARG DT_ONEAGENT_OPTIONS="flavor=default&include=<technology1>&include=<technology2>" ENV DT_HOME="/opt/dynatrace/oneagent" RUN mkdir -p "$DT_HOME" && \ wget -O "$DT_HOME/oneagent.zip" "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_API_TOKEN&$DT_ONEAGENT_OPTIONS" && \ unzip -d "$DT_HOME" "$DT_HOME/oneagent.zip" && \ rm "$DT_HOME/oneagent.zip" ENV LD_PRELOAD="/opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so"
Note: The commands above that use
wget
andunzip
might fail if they aren't provided by the base image. -
Build your application image.
In an OpenShift context the above Dockerfile could be used for binary builds as follows:
oc new-build --binary --strategy=docker --allow-missing-images yourapp oc patch bc/yourapp --type=json --patch='[{"op":"remove","path":"/spec/strategy/dockerStrategy/from"}]' oc start-build yourapp --from-dir=. --follow
You can monitor your application containers with a different Dynatrace environment.
For OneAgent version 1.139 or higher, if you have an existing application image where you have already added the OneAgent code modules for a specific Dynatrace environment, you can have the OneAgent report to another Dynatrace environment without rebuilding your application image.
For this, you need to make a call to the REST endpoint of your second Dynatrace environment. Don't forget to adapt the respective placeholders<environmentID>
and<token>
.curl "https://<environmentID>.live.dynatrace.com/api/v1/deployment/installer/agent/connectioninfo?Api-Token=<token>"
In return, you get a JSON object that covers the required information that needs to be passed as an environment variable to the application container.
Make sure you set the environment variables of the application container as described below:DT_TENANT
: equalstenantUUID
DT_TENANTTOKEN
: equalstenantToken
DT_CONNECTION_POINT
: semi-colon separated list ofcommunicationEndpoints
- optional Configure network zones
You can configure network zones as an environment variable:
DT_NETWORK_ZONE
: equalsyour.network.zone
See network zones for more information.
-
optional Configure a proxy address
In case you run an environment with proxy, you need to set the
DT_PROXY
environment variable in the application container to pass the proxy credentials to OneAgent.Note: For Alpine Linux-based containers, you might need to update the
wget
shipped with the Alpine image to allow for proxy authentication for the download of OneAgent.
- Set the following build-time variables:
$DT_API_URL
(The API URL of your Dynatrace environment)$DT_PAAS_TOKEN
(The PaaS token to download the code modules)$DT_ONEAGENT_TECHNOLOGY
(The module that is downloaded, for examplephp
)
- Add the following commands to the Dockerfile:
RUN mkdir -p /opt/dynatrace/oneagent && ARCHIVE=$(mktemp) && wget -O $ARCHIVE "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_PAAS_TOKEN&flavor=default&arch=arm&include=$DT_ONEAGENT_TECHNOLOGY" && unzip -o -d /opt/dynatrace/oneagent $ARCHIVE && rm -f $ARCHIVE
ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
Update
You need to manually update OneAgent by rebuilding the container image every time a new version of a code module is needed.
Uninstall
To uninstall OneAgent from application-only monitoring
-
Remove the two lines of code from the application image.
COPY --from=<ACTIVEGATE-ADDRESS>/linux/oneagent-codemodules:<TECHNOLOGY> / / ENV LD_PRELOAD /opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so
-
Rebuild the application image.
docker build -t yourapp .
-
Remove the following commands from your Dockerfile.
ARG DT_API_URL="https://<environmentID>.live.dynatrace.com/api" ARG DT_API_TOKEN="<token>" ARG DT_ONEAGENT_OPTIONS="flavor=default&include=<technology1>&include=<technology2>" ENV DT_HOME="/opt/dynatrace/oneagent" RUN mkdir -p "$DT_HOME" && \ wget -O "$DT_HOME/oneagent.zip" "$DT_API_URL/v1/deployment/installer/agent/unix/paas/latest?Api-Token=$DT_API_TOKEN&$DT_ONEAGENT_OPTIONS" && \ unzip -d "$DT_HOME" "$DT_HOME/oneagent.zip" && \ rm "$DT_HOME/oneagent.zip" ENTRYPOINT [ "/opt/dynatrace/oneagent/dynatrace-agent64.sh" ] CMD [ "executable", "param1", "param2" ] # the command of your application, for example, Java
-
Rebuild the application image.
docker build -t yourapp .