Containerized, auto-scalable private Synthetic locations on Kubernetes

Dynatrace version 1.264+

Containerized, auto-scalable private Synthetic locations on Kubernetes and its commercial distribution OpenShift are an alternative to deploying Synthetic-enabled ActiveGates on separate hosts or virtual machines and then assigning them to private locations for the execution of synthetic monitors.

Unlike individual Synthetic-enabled ActiveGates that are deployed and assigned to private locations (and then tracked via utilization metrics) containerized locations are deployed as a whole, with a minimum and maximum number of ActiveGates as the necessary input parameters.

Kubernetes and OpenShift aren't just additional supported ActiveGate platforms along with Windows and Linux; with this offering, containerized private Synthetic locations:

• Are auto-scalable (based on utilization metrics and the maximum/minimum number of ActiveGates specified).

• Are easy to manage and maintain.
• Support the synthetic monitoring of cloud-native solutions that require container-based application development.
• Can be deployed faster while minimizing downtime.
• Are automatically tracked for resource utilization as a part of auto-scaling operations.

You can execute scheduled as well as on-demand executions of all types on synthetic monitors on containerized locations.

You can manage Kubernetes/OpenShift locations via the Dynatrace web UI and the existing Synthetic - Locations, nodes, and configuration API v2. Additional Early Adopter endpoints in this API facilitate the deployment of Kubernetes locations; the new endpoints help you generate the commands that need to be executed on the Kubernetes cluster.

Architecture

Containerized private Synthetic locations are deployed as a whole.

• Each location has multiple Synthetic-enabled ActiveGates configured as pods. During location creation, you specify a minimum and maximum number of ActiveGates when setting up a location.

• The StatefulSet is considered as the location.

  You can have one or more locations per namespace. See Requirements and Recommendations and caveats below.

  You can have one or more auto-scalable locations per Kubernetes cluster.

Locations are auto-scaled by adjusting the number of ActiveGates per location by the following additional parts of the containerized location architecture.

• The Synthetic metric adapter requests and receives utilization metrics for the containerized ActiveGates from the Dynatrace Cluster.

  There is one Synthetic metric adapter per Kubernetes cluster.

  The metric adapter is configured to communicate with a single Dynatrace environment.

  Installing a Synthetic metric adapter requires super-user roles in Kubernetes—see Install a containerized location below.

• The horizontal pod auto-scaler scales a location by adjusting the number of ActiveGates based on the utilization data it receives from the Synthetic metric adapter.

  There is one horizontal pod auto-scaler per location.

Requirements

Containerized private Synthetic locations are supported with Dynatrace version 1.264+ on Kubernetes 1.22–1.25 with persistent volume and kubectl support.

• Additional support for Kubernetes 1.26+ is available in the installation workflow.
• All kinds of Kubernetes implementations are supported, whether cloud or local (for example, Amazon EKS or Minikube).
• OpenShift versions compatible with the supported Kubernetes versions are supported.

The ActiveGate hardware requirements below are listed by size.

Best practices and caveats

ActiveGates

• We recommend the S ActiveGate size and a minimum of two ActiveGates per location.
• All ActiveGates within a location are always the same size.
• Once specified, ActiveGate size for a location can't be changed because persistent storage can't be resized.
• Kubernetes locations follow the same rules as other locations in that an ActiveGate can't be added to multiple locations simultaneously.
• You cannot combine containerized and non-containerized ActiveGates in the same location.

Locations

• We recommend installing each location in its own namespace.

• If deploying more than one location per namespace, use different names for the respective ActiveGate resources—see Install a containerized location below.

• Locations that share a single Kubernetes namespace must be connected to the same Dynatrace environment as the Synthetic metric adapter in order to be auto-scalable. For example, assume that Location A and the metric adapter are configured for Environment X. However, Location A shares a namespace with Location B, which is configured for Environment Y. In such a case, Location A is auto-scalable; Location B is not auto-scalable.

• If you want to install a location in the same namespace as other Dynatrace resources such as Dynatrace Operator, be aware of the more demanding hardware and system requirements for containerized Synthetic-enabled ActiveGates.

Synthetic metric adapter

• The best practice is to deploy the Synthetic metric adapter in its own namespace per Kubernetes cluster. The Synthetic metric adapter can share a namespace with a location. However, deploying the metric adapter in its own namespace ensures that it isn't deleted when a location is taken down.
• The metric adapter can only communicate with a single Dynatrace environment, so location auto-scaling works just for that environment.
• The image for the Synthetic metric adapter is only available in a public registry.

Auto-scaling specifics

For auto-scaling purposes, the Synthetic metric adapter needs access to and extends the Kubernetes API by specifying a new API service—v1beta1.external.metrics.k8s.io.

This API service is defined in the Synthetic metric adapter template—see below.

yamlcopydownload
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: dynatrace-metrics-apiserver
    namespace: {{adapterNamespace}}
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

Note that any other metric adapter, such as a Prometheus adapter, in the Kubernetes cluster might also use this service.

The Synthetic metric adapter also modifies an existing resource in its template—the horizontal-pod-autoscaler ServiceAccount in kube-system namespace.

yamlcopydownload
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hpa-controller-dynatrace-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dynatrace-metrics-server-resources
subjects:
  - kind: ServiceAccount
    name: horizontal-pod-autoscaler
    namespace: kube-system

Install a containerized location

Set up and manage a containerized location in the Dynatrace web UI at Settings > Web and Mobile monitoring > Private Synthetic locations.

Initial setup for a Kubernetes/OpenShift location

Deploy the location

Deploy the Synthetic metric adapter

Initial setup for a Kubernetes/OpenShift location

1. Select Add Kubernetes location or Add OpenShift location on the Private Synthetic locations page.

2. Provide a Location name of your choice.

3. Select a Geographic location, for example, San Francisco, California, United States. (Note that you cannot Save changes until you've specified a name and location.)

4. In the ActiveGates section:

   1. Specify a Minimum and Maximum number of ActiveGates for your location. These settings are the auto-scaling parameters that the horizontal pod auto-scaler uses (manage-private-locations#location-details).
   2. Select an ActiveGate Node size (XS, S, or M). See also Requirements and Best practices and caveats.
   3. The Deployment platform is preselected based on your selection of Kubernetes or OpenShift.

5. Kubernetes only If your Kubernetes implementation is based on a later release than 1.21–1.25, turn on Use Kubernetes version 1.26+. See also Requirements and Best practices and caveats.

   If you change this setting after downloading the location template, you need to repeat the deployment procedure.

6. optional Turn on problem generation if all ActiveGates go offline.

7. Save changes before proceeding with deploying the location and metric adapter.

Your named location is displayed on the Private Synthetic locations page with the Kubernetes or OpenShift logo. Note that no ActiveGates are assigned to the location at this point.

Deploy the location

Select your location in Private Synthetic locations to download the location template and generate the commands that need to be executed on the Kubernetes cluster.

1. In the Deployment section, create a PaaS token (Create token) or paste an existing token. The PaaS token is required to generate ActiveGate connection tokens for communication with your Dynatrace environment.

   Existing tokens are listed on the Access tokens page. Note that a PaaS token is only displayed once upon creation, after which it's stored encrypted and can't be revealed. We recommend storing your PaaS token in a password manager so that you can reuse it for creating additional private locations within your Kubernetes cluster.

2. Provide an ActiveGate name or use the default. This name is used as the prefix for ActiveGates deployed as part of the location. The first ActiveGate is named <prefix>-0, the second ActiveGate <prefix>-1, and so on. This name is also used as the StatefulSet name.

3. Provide a Location namespace name or use the default. (Leave the Metric adapter namespace as is. This field is only necessary for generating the template for the Synthetic metric adapter.)

   The Download synthetic.yaml button is enabled after you provide a PaaS token, ActiveGate name, and location namespace name.

   The values of fields in the Deployment section are not persistent. If you navigate away from the page, you need to re-enter the values.

   

4. Select Download synthetic.yaml. This is the location template file. You can rename the file to match your location for easy identification.

5. Copy the downloaded location template over to your Kubernetes cluster.

6. Copy and execute the generated commands on your Kubernetes cluster. Your PaaS token is automatically appended to the commands displayed.

   Execute the commands from the same location as the template file.

   If you've renamed the template file, use the new filename in the commands.

   

7. optional Run the following command to list all pods in a given namespace (dynatrace in the sample below) and verify their deployment.

   shellcopydownload
   kubectl get pod -n dynatrace

   You can also view pods on the Deployment status > ActiveGates page by filtering with the key-value pairs Running in container: True and With modules: Synthetic.

   

Deploy the Synthetic metric adapter

This procedure generates a separate template for the Synthetic metric adapter. You then execute generated commands on your Kubernetes cluster to deploy the metric adapter.

1. In the Deployment section, create a Metrics token (Create token) or paste an existing token. The metric token is an access token for fetching utilization data from Dynatrace. Existing tokens are listed on the Access tokens page.

2. Provide a Metric adapter namespace name or use the default. (Leave the Location namespace and ActiveGate name as is. These fields are only necessary for generating the template for the location.)

   The Download synthetic-adapter.yaml button is enabled after you provide a metric token and metric adapter namespace name.

   The values of fields in the Deployment section are not persistent. If you navigate away from the page, you need to re-enter the values.

   

3. Select Download synthetic-adapter.yaml. This is the template file for the Synthetic metric adapter.

4. Copy the downloaded metric adapter template over to your Kubernetes cluster.

5. Copy and execute the generated commands on your Kubernetes cluster. Your metric token is automatically appended to the commands displayed.

   Execute the commands from the same location as the template file.

   If you've renamed the template file, use the new filename in the commands.

   

Update a containerized location or its ActiveGates

Any updates to a location require editing or regenerating the location template file and then applying the changes via kubectl.

To update ActiveGate versions

1. In the location template file:

   1. Edit the activegate container image image: <environment-id>.live.dynatrace.com/linux/activegate:latest to provide a new built unit version as shown below.

      ymlcopydownload
      containers:
            - name: activegate
              image: myenvironmentid.dynatrace.com/linux/activegate:1.263.14
              imagePullPolicy: Always

   2. Edit the synthetic container image image: <environment-id>.dynatrace.com/linux/dynatrace-synthetic:latest to provide a new Synthetic build unit version as shown below.

      ymlcopydownload
      - name: synthetic
              image: myenvironmentid.dynatrace.com/linux/dynatrace-synthetic:1.264.6
              imagePullPolicy: Always

2. Execute the following command to apply the changes on your Kubernetes cluster. Be sure to use your location template filename in place of synthetic.yaml.

   shellcopydownload
   kubectl apply -f ./synthetic.yaml

Any update redeploys ActiveGates in the reverse order of their deployment. For example, if your location contains the ActiveGates activegate-name-0 and activegate-name-1, activegate-name-1 is stopped and redeployed first.

The redeployed ActiveGate pod uses the same persistent volume deployed for log continuity.

Delete a location or metric adapter on Kubernetes

The commands generated when deploying a location and the Synthetic metric adapter also include code snippets for deleting them on Kubernetes. You may copy and store these commands for future reference.

At any point, you can regenerate the commands for the respective namespaces.

Location

1. Select your location in Private Synthetic locations.

2. Re-enter the Paas token, ActiveGate name, and Location namespace name.

3. Copy and use the location Cleanup commands.

   

Note that this procedure only deletes Kubernetes resources; it doesn't delete the location you initially set up in Dynatrace.

Synthetic metric adapter

1. Select your location in Private Synthetic locations.

2. Re-enter the Metrics token and Metric adapter namespace name.

3. Copy and use the metric adapter Cleanup command.

   

Proxy configuration

Add the following code at the top of your location template file to insert a ConfigMap resource containing your proxy server information.

In the code sample below:

• The proxy server is used for connections to the Dynatrace Cluster and tested resources.
• The namespace (namespace: dynatrace) must be the location namespace.

ymlcopydownload
kind: ConfigMap
apiVersion: v1
data:
  custom.properties: |-
    [http.client]
    proxy-server = 10.102.43.210
    proxy-port = 3128
    proxy-user = proxyuser
    proxy-password = proxypass
metadata:
  name: ag-custom-configmap
  namespace: dynatrace
---

Add the following code at template.volumes.

ymlcopydownload
- name: ag-custom-volume
          configMap:
            name: ag-custom-configmap
            items:
              - key: custom.properties
                path: custom.properties

Add the following code to the ActiveGate container configuration under volumeMounts:.

ymlcopydownload
- name: ag-custom-volume
              mountPath: /var/lib/dynatrace/gateway/config_template/custom.properties
              subPath: custom.properties

Non-scalable containerized locations

Auto-scalable locations become non-scalable for any of the following reasons.

• The location reaches its maximum number of pods in the StatefulSet, and location utilization is over the threshold of 80%. No new ActiveGate pods are created until the maximum number of ActiveGates is increased.

• The Synthetic metric adapter stops working, and the location horizontal pod auto-scalers don't receive the metrics required for auto-scaling.

  You can run the following command to verify the state of a pod auto-scaler. In the example below, dynatrace is the location namespace.

  shellcopydownload
  kubectl describe hpa -n dynatrace

  If ScalingActive is set to False in the output, the auto-scaler isn't receiving metric data.

API: Synthetic - Locations, nodes, and configuration API v2

You can automate the deployment and manage containerized locations via the existing Synthetic - Locations, nodes, and configuration API v2. Early Adopter endpoints added to this API to facilitate the deployment of Kubernetes locations. The new endpoints help generate the commands you need to execute on the Kubernetes cluster.

• The GET location YAML endpoint (/synthetic/locations/{LocationId}/yaml) fetches the location template file based on the location ID of the location you initially set up for containerized deployment.
• The GET apply commands endpoint (synthetic/locations/commands/apply) fetches the list of commands to deploy a location on Kubernetes/Openshift.
• The GET delete commands endpoint (synthetic/locations/{LocationId}/commands/delete)fetches the commands to delete a containerized location.