Set up the Dynatrace GCP metric and log integration on a new GKE Autopilot cluster

Dynatrace version 1.230+

Follow the instructions below to set up Google Cloud Platform monitoring for metrics and logs on a new GKE Autopilot cluster, using Google Cloud Shell. During setup, a new Pub/Sub subscription will be created. GKE will run two containers: a metric forwarder and a log forwarder. After installation, you'll get metrics, logs, dashboards, and alerts for your configured services in Dynatrace.

If you prefer to run the deployment script on an existing standard GKE or GKE Autopilot cluster, see Set up the Dynatrace GCP log and metric integration on an existing GKE cluster.

For other deployment options, see Alternative deployment scenarios.

This page describes how to install version 1.0 of the GCP integration on a GKE cluster.

If you already have an earlier version installed, you need to migrate.

Limitations

Dynatrace GCP log integration supports maximum 2 GB of data processing per hour.

Dynatrace GCP metric integration supports up to 50 GCP projects with the standard deployment. To monitor larger environments, you need to enable metrics scope. See Monitor multiple GCP projects - Large environments.

Prerequisites

To deploy the integration, you need to make sure the following requirements are met:

Linux OS only
Dynatrace environment access You need to configure the Dynatrace endpoint (environment, cluster or ActiveGate url) to which the GKE autopilot cluster will send metrics/logs. Make sure that you have a direct network access or, in case of any proxy or other component present in between, the communication is not affected.

GCP permissions

Running the deployment script requires a list of permissions. You can create a custom role (see below) and use it to deploy dynatrace-gcp-monitor.

Create a YAML file named dynatrace-gcp-monitor-helm-deployment-role.yaml with the following content:

dynatrace-gcp-monitor-helm-deployment-role.yaml

yaml
title: Dynatrace GCP Monitor helm deployment role
description: Role for Dynatrace GCP Monitor helm and pubsub deployment
stage: GA
includedPermissions:
  - container.clusters.get
  - container.configMaps.create
  - container.configMaps.delete
  - container.configMaps.get
  - container.configMaps.update
  - container.deployments.create
  - container.deployments.delete
  - container.deployments.get
  - container.deployments.update
  - container.namespaces.create
  - container.namespaces.get
  - container.pods.get
  - container.pods.list
  - container.replicaSets.create
  - container.replicaSets.get
  - container.replicaSets.getScale
  - container.replicaSets.getStatus
  - container.replicaSets.list
  - container.secrets.create
  - container.secrets.delete
  - container.secrets.get
  - container.secrets.list
  - container.secrets.update
  - container.serviceAccounts.create
  - container.serviceAccounts.delete
  - container.serviceAccounts.get
  - container.services.create
  - container.services.delete
  - container.services.get
  - container.statefulSets.create
  - container.statefulSets.delete
  - container.statefulSets.get
  - container.statefulSets.update
  - iam.roles.create
  - iam.roles.list
  - iam.roles.update
  - iam.serviceAccounts.actAs
  - iam.serviceAccounts.create
  - iam.serviceAccounts.getIamPolicy
  - iam.serviceAccounts.list
  - iam.serviceAccounts.setIamPolicy
  - pubsub.subscriptions.create
  - pubsub.subscriptions.get
  - pubsub.subscriptions.list
  - pubsub.topics.attachSubscription
  - pubsub.topics.create
  - pubsub.topics.getIamPolicy
  - pubsub.topics.list
  - pubsub.topics.setIamPolicy
  - pubsub.topics.update
  - resourcemanager.projects.get
  - resourcemanager.projects.getIamPolicy
  - resourcemanager.projects.setIamPolicy
  - serviceusage.services.enable
  - serviceusage.services.get

Run the command below, replacing <your_project_ID> with the project ID where you want to deploy the Dynatrace GCP integration.

bash
gcloud iam roles create dynatrace_monitor.helm_deployment --project=<your_project_ID> --file=dynatrace-gcp-monitor-helm-deployment-role.yaml

Note: Be sure to add this role to your GCP user. For details, see Grant or revoke a single role.

Configure log export

Run the following shell script in the GCP project you've selected for deployment.

Note: Be sure to replace <your-subscription-name> and <your-topic-name> with your own values.

plaintext
wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-monitor/master/scripts/deploy-pubsub.sh
chmod +x deploy-pubsub.sh
./deploy-pubsub.sh --topic-name <your-topic-name> --subscription-name <your-subscription-name>

Configure log export to send the desired logs to the GCP Pub/Sub topic created above.

Dynatrace permissions

You need to create a token with a set of permissions.

In the Dynatrace menu, go to Access tokens.
Select Generate new token.
Enter a name for your token.
Under Template, select GCP Services Monitoring.
Select Generate.
Copy the generated token to the clipboard. Store the token in a password manager for future use.

Alternatively, you can create the token and add permissions manually.

Add manually

Create an API token and enable the following permissions:

API v1:
- Read configuration
- Write configuration
API v2:
- Ingest metrics
- Read extensions
- Write extensions
- Read extensions monitoring configuration
- Write extensions monitoring configuration
- Read extensions environment configuration
- Write extensions environment configuration
- Ingest logs

Log ingestion

If you are using Log Monitoring v1, enable the latest version of Dynatrace Log Monitoring.
Determine where log ingestion will be performed, according to your deployment. This distinction is important when configuring the parameters for this integration.
- For SaaS deployments: SaaS log ingest, where log ingestion is performed directly through the Cluster API. recommended
- For Managed deployments: You can use an existing ActiveGate for log ingestion. For information on how to deploy it, see ActiveGate installation.

Install

Complete the steps below to finish your setup.

Download the Helm deployment package in Google Cloud Shell

Configure parameter values

Run the deployment script

Download the Helm deployment package in Google Cloud Shell

bash
wget -q "https://github.com/dynatrace-oss/dynatrace-gcp-monitor/releases/latest/download/helm-deployment-package.tar"; tar -xvf helm-deployment-package.tar; chmod +x helm-deployment-package/deploy-helm.sh

Configure parameter values

The Helm deployment package contains a values.yaml file with the necessary configuration for this deployment. Go to helm-deployment-package/dynatrace-gcp-monitor and edit the values.yaml file, setting the required and optional parameter values as follows.

Note: You might want to store this file somewhere for future updates, since it will be needed in case of redeployments. Also, keep in mind that its schema can change. In such case, you should use the new file and only copy over the parameter values.

Parameter name	Description	Default value
`gcpProjectId`	required The ID of the GCP project you've selected for deployment.	Your current project ID
`deploymentType`	required Leave to 'all'.	`all`
`dynatraceAccessKey`	required Your Dynatrace API token with the required permissions.
`dynatraceUrl`	required For SaaS log/metric ingestion, it's your environment URL (`https://<your-environment-id>.live.dynatrace.com`). For Managed log/metric ingestion, it's your cluster URL (`https://<cluster_ID>.managed.internal.dynatrace/e/<your_environment_ID>`). For Managed log/metric ingestion with an existing ActiveGate, it's the URL of your ActiveGate (`https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID>`). Note: To determine `<your-environment-id>`, see environment ID.
`logsSubscriptionId`	required The ID of your log Sink Pub/Sub subscription. For details, see Configure log export.
`dynatraceLogIngestUrl`	optional You can set it if you want to ingest logs separately from metrics. For SaaS log ingestion, it's your environment URL (`https://<your_environment_ID>.live.dynatrace.com`) For Managed log ingestion with an existing ActiveGate, it's the URL of your ActiveGate (`https://<your_activegate_IP_or_hostname>:9999/e/<your_environment_ID>`) Note: To determine `<your-environment-id>`, see environment ID.
`requireValidCertificate`	optional If set to `true`, Dynatrace requires the SSL certificate of your Dynatrace environment. For SaaS log ingestion, we recommend leaving the default value. For Managed log ingestion with a new ActiveGate, we recommend setting this value to `false`.	`true`
`selfMonitoringEnabled`	optional Send custom metrics to GCP to quickly diagnose if `dynatrace-gcp-monitor` processes and sends metrics/logs to Dynatrace properly. For details, see Self-monitoring metrics for the Dynatrace GCP integration.	`false`
`serviceAccount`	optional Name of the service account to be created.
`dockerImage`	optionalDynatrace GCP Monitor docker image. We recommend using the default value, but you can adapt it if needed.	`dynatrace/dynatrace-gcp-monitor:v1-latest`
`logIngestContentMaxLength`	optional The maximum content length of a log event. Should be the same as or lower than the setting on your Dynatrace environment.	`8192`
`logIngestAttributeValueMaxLength`	optional The maximum length of the log event attribute value. If it exceeds the server limit, content will be truncated.	`250`
`logIngestRequestMaxEvents`	optional The maximum number of log events in a single payload to the logs ingestion endpoint. If it exceeds the server limit, payload will be rejected with code `413`.	`5000`
`logIngestRequestMaxSize`	optional The maximum size in bytes of a single payload to the logs ingestion endpoint. If it exceeds the server limit, payload will be rejected with code `413`.	`1048576`
`logIngestEventMaxAgeSeconds`	optional Determines the maximum age of a forwarded log event. Should be the same as or lower than the setting on your Dynatrace environment.	`86400`
`printMetricIngestInput`	optional If set to `true`, the GCP Monitor outputs the lines of metrics to stdout.	`false`
`serviceUsageBooking`	optional Service usage booking is used for metrics and determines a caller-specified project for quota and billing purposes. If set to `source`, monitoring API calls are booked in the project where the Kubernetes container is running. If set to `destination`, monitoring API calls are booked in the project that is monitored. For details, see Monitor multiple GCP projects - Standard environments - Step 4.	`source`
`useProxy`	optional Depending on the value you set for this flag, the GCP Monitor will use the following proxy settings: Dynatrace (set to `DT_ONLY`), GCP API (set to `GCP_ONLY`), or both (set to `ALL`).	By default, proxy settings are not used.
`httpProxy`	optional The proxy HTTP address; use this flag in conjunction with `USE_PROXY`.
`httpsProxy`	optional The proxy HTTPS address; use this flag in conjunction with `USE_PROXY`.
`gcpServicesYaml`	optional Configuration file for GCP services.
`queryInterval`	optional Metrics polling interval in minutes. Allowed values: `1` - `6`	`3`
`vpcNetwork`	optional Existing VPC Network where the autopilot cluster will be deployed. Shared VPC is not supported.	`default`
`scopingProjectSupportEnabled`	optional Set to `true` when metrics scope is configured, so metrics will be collected from all projects added to the metrics scope. For details, see Monitor multiple GCP projects - Large environments.	`false`

Choose which services you want Dynatrace to monitor.

By default, the Dynatrace GCP integration starts monitoring a set of selected services. Uncomment any additional services you want Dynatrace to monitor in the values.yaml file.

Note: For DDU consumption information, see Monitoring consumption.

Run the deployment script

The deployment script will automatically create the new GKE Autopilot cluster named dynatrace-gcp-monitor and deploy the installation to it. The latest versions of GCP extensions will be uploaded.

bash
cd helm-deployment-package
./deploy-helm.sh --create-autopilot-cluster

Note: To set a different name for the new cluster, run the command below instead, making sure to replace the placeholder (<name-of-new-cluster>) with your preferred name.

bash
cd helm-deployment-package
./deploy-helm.sh --create-autopilot-cluster --autopilot-cluster-name <name-of-new-cluster>

Note: To keep the existing versions of present extensions and install the latest versions for the rest of the selected extensions, if they are not present, run the command below instead.

bash
cd helm-deployment-package
./deploy-helm.sh --create-autopilot-cluster --without-extensions-upgrade

Verify installation

To check whether installation was successful

Check if the container is running.

Note: After the installation, it may take couple of minutes until the container is up and running.
```
plaintext
kubectl -n dynatrace get pods
```
Check the container logs for errors or exceptions. You have two options:
In the Kubernetes CLIIn your GCP console
Run the following commands.
```
plaintext
kubectl -n dynatrace logs -l app=dynatrace-gcp-monitor -c dynatrace-gcp-monitor-metrics
kubectl -n dynatrace logs -l app=dynatrace-gcp-monitor -c dynatrace-gcp-monitor-logs
```
To check the container logs for errors in your GCP console
1. Go to Logs explorer.
2. Use the filters below to get metric and/or log ingest logs from the Kubernetes container:
- resource.type="k8s_container"
- resource.labels.container_name="dynatrace-gcp-monitor-metrics" (for metric ingest logs)
- resource.labels.container_name="dynatrace-gcp-monitor-logs" (for log ingest logs)
Check if dashboards are imported.

In the Dynatrace menu, go to Dashboards and filter by Tag for Google Cloud. A number of dashboards for Google Cloud Services should be available.

Enable alerting

To activate alerting, you need to enable metric events for alerting in Dynatrace.

To enable metric events

In the Dynatrace menu, go to Settings.
In Anomaly detection, select Metric events.
Filter for GCP alerts and turn on On/Off for the alerts you want to activate.

View metrics and logs

After deploying the integration, you can:

See metrics from monitored services: in the Dynatrace menu, go to Metrics and filter by gcp.
View and analyze GCP logs: in the Dynatrace menu, go to Logs and, to look for GCP logs, filter by cloud.provider: gcp.

View enabled services

The list of currently enabled services can be found in the cluster's ConfigMap named dynatrace-gcp-monitor-config.

Update services

Adding, removing, and updating versions of existing services is done by modifying the corresponding list of services and redeploying. To do that, follow the steps below.

Edit values.yaml by commenting or uncommenting configuration blocks corresponding to specific services.

Note: If you already deleted the deployment package and don't have the original values.yaml file anymore, you can use a new one. In this case, the new file will override your previous configuration, so make sure not to accidentally disable monitoring of previously monitored services. Terminology within the file includes:
- service: represents GCP service name you want to monitor. Services are grouped by extensions, but you can decide what you need to monitor on a lower level (featureSets)
- featureSet: a set of metrics for a given service. default_metrics is a default featureSet with a recommended set of metrics to be monitored. In more specific use cases, you can consider monitoring such sets as istio featureSet for gae_instance service
- filter_conditions: a service-level filter that enables you to narrow the monitoring scope. It is based on the GCP Monitoring filters.
  Example:
```
yaml
filter_conditions:
resource.labels.location = "us-central1-c" AND resource.labels.namespace_name = "dynatrace"
```
Update monitored services by running the script below.

Note: Version upgrade of extensions is done by default. To keep the versions of existing extensions, run the script with the --without-extensions-upgrade parameter.
```
bash
cd helm-deployment-package
./deploy-helm.sh
```
If you removed services from monitoring, find the relevant extensions in Dynatrace Hub (in the Dynatrace menu, go to Extensions) and delete them to remove service-specific assets (dashboards, alerts, etc).

Example

In the following example

The gae_instance service is disabled.
For the gce_instance service, only two feature sets are enabled: default_metrics and istio.

yaml
# Google App Engine Instance
    #- service: gae_instance
    #  featureSets:
    #    - default_metrics
    #  vars:
    #    filter_conditions: ""
    # Google VM Instance
    - service: gce_instance
      featureSets:
        - default_metrics
    #    - agent
    #    - firewallinsights
        - istio
    #    - uptime_check
      vars:
        filter_conditions: ""

For a complete list of the GCP supported services, see Google Cloud Platform supported service metrics.

Change deployment settings

To change the deployment type (all, metrics, or logs), see Change deployment type.
To change which services are monitored, see Add or remove services.
To change other settings in values.yaml, see Change parameters from values.yaml.

Change parameters from `values.yaml`

To load a new values.yaml file, you need to upgrade your Helm release.

To update your Helm release

Find out what Helm release version you're using.
```
plaintext
helm ls -n dynatrace
```
Run the command below, making sure to replace <your-helm-release> with value from previous step.
```
plaintext
helm upgrade <your-helm-release> dynatrace-gcp-monitor -n dynatrace
```

For details, see Helm upgrade.

Change deployment type

To change the deployment type (all, metrics, or logs)

Find out what helm release version you're using.
```
plaintext
helm ls -n dynatrace
```
Uninstall the release.

Note: Be sure to replace <your-helm-release> with the release name from the previous output.
```
plaintext
helm uninstall <your-helm-release> -n dynatrace
```
Edit deploymentType in values.yaml with the new value and save the file.
Run the deployment command again. For details, see Run the deployment script.

Troubleshoot

To investigate potential deployment and connectivity issues

Verify installation
Enable self-monitoring optional
Check the dynatrace_gcp_<date_time>.log log file created during the installation process.

This file will be created each time the installation script runs.
The debug information won't contain sensitive data such as the Dynatrace access key.

If you are contacting Dynatrace ONE:
- Make sure to provide the dynatrace_gcp_<date_time>.log log file described in the previous step.
- Provide version information.
  - For issues during installation, check the version.txt file.
  - For issues during runtime, check container logs.

Uninstall

Find out what Helm release version you're using.

plaintext
helm ls -n dynatrace

Uninstall the release.

Note: Be sure to replace <your-helm-release> with the release name from the previous output.

plaintext
helm uninstall <your-helm-release> -n dynatrace

Alternatively, you can delete the namespace.

bash
kubectl delete namespace dynatrace

To remove all monitoring assets (dashboards, alerts, etc) from Dynatrace, you need to remove all GCP extensions.

To remove an extension

In the Dynatrace menu, go to Extensions and search for the GCP extensions.
Select an extension you want to remove, and then select the trash icon in the Actions column to remove it.

Repeat the procedure until you remove all GCP extensions.

Monitoring consumption

Metric ingestion

All cloud services consume DDUs. The amount of DDU consumption per service instance depends on the number of monitored metrics and their dimensions (each metric dimension results in the ingestion of 1 data point; 1 data point consumes 0.001 DDUs). For details, see Extending Dynatrace (Davis data units).

Log ingestion

DDU consumption applies to cloud Log Monitoring. See DDUs for Log Monitoring for details.

Set up the Dynatrace GCP metric and log integration on a new GKE Autopilot cluster

Limitations

Prerequisites

GCP permissions

Configure log export

Dynatrace permissions

Log ingestion

Install

Download the Helm deployment package in Google Cloud Shell

Configure parameter values

Run the deployment script

Verify installation

Enable alerting

View metrics and logs

View enabled services

Update services

Example

Change deployment settings

Change parameters from values.yaml

Change deployment type

Troubleshoot

Uninstall

Monitoring consumption

Metric ingestion

Log ingestion

Change parameters from `values.yaml`