Set up the Dynatrace GCP log and metric integration on an existing GKE cluster

As an alternative to the main deployment, where the deployment script runs in a new automatically created GKE Autopilot cluster, you can choose to run the deployment script on an existing standard GKE or GKE Autopilot cluster. In this scenario, you will set up Google Cloud Platform monitoring for metrics and logs in Google Cloud Shell. During setup, a new Pub/Sub subscription will be created. GKE will run two containers: a metric forwarder and a log forwarder. After installation, you'll get metrics, logs, dashboards, and alerts for your configured services in Dynatrace.

For other deployment options, see Alternative deployment scenarios.

Prerequisites

To deploy the integration, you need to make sure the following requirements are met:

GCP permissions

Running the deployment script requires a list of permissions. You need to create a custom role (see below) and use it to deploy dynatrace-gcp-function.

1. Create a YAML file named dynatrace-gcp-function-helm-deployment-role.yaml with the following content:

yamlcopydownload
title: Dynatrace GCP Function helm deployment role
description: Role for Dynatrace GCP function helm and pubsub deployment
stage: GA
includedPermissions:
  - container.clusters.get
  - container.configMaps.create
  - container.configMaps.delete
  - container.configMaps.get
  - container.configMaps.update
  - container.deployments.create
  - container.deployments.delete
  - container.deployments.get
  - container.deployments.update
  - container.namespaces.create
  - container.namespaces.get
  - container.pods.get
  - container.pods.list
  - container.replicaSets.create
  - container.replicaSets.get
  - container.replicaSets.getScale
  - container.replicaSets.getStatus
  - container.replicaSets.list
  - container.secrets.create
  - container.secrets.delete
  - container.secrets.get
  - container.secrets.list
  - container.secrets.update
  - container.serviceAccounts.create
  - container.serviceAccounts.delete
  - container.serviceAccounts.get
  - container.services.create
  - container.services.delete
  - container.services.get
  - container.statefulSets.create
  - container.statefulSets.delete
  - container.statefulSets.get
  - container.statefulSets.update
  - iam.roles.create
  - iam.roles.list
  - iam.roles.update
  - iam.serviceAccounts.actAs
  - iam.serviceAccounts.create
  - iam.serviceAccounts.getIamPolicy
  - iam.serviceAccounts.list
  - iam.serviceAccounts.setIamPolicy
  - pubsub.subscriptions.create
  - pubsub.subscriptions.get
  - pubsub.subscriptions.list
  - pubsub.topics.attachSubscription
  - pubsub.topics.create
  - pubsub.topics.getIamPolicy
  - pubsub.topics.list
  - pubsub.topics.setIamPolicy
  - pubsub.topics.update
  - resourcemanager.projects.get
  - resourcemanager.projects.getIamPolicy
  - resourcemanager.projects.setIamPolicy
  - serviceusage.services.enable
  - serviceusage.services.get

2. Run the command below, replacing <your_project_ID> with the project ID where you want to deploy the Dynatrace integration.

bashcopydownload
gcloud iam roles create dynatrace_function.helm_deployment --project=<your_project_ID> --file=dynatrace-gcp-function-helm-deployment-role.yaml

Note: Be sure to add this role to your GCP user. For details, see Grant or revoke a single role.

GCP settings

The location where you deploy the integration determines whether you need make any additional settings.

Deploy on an existing GKE Autopilot cluster

If you deploy the integration on an existing GKE Autopilot cluster, you don't need to make any additional settings.

Deploy on an existing GKE standard cluster

If you deploy the integration on an existing GKE standard cluster, you need to

• Enable Workload Identity on a cluster
• Enable GKE_METADATA on the GKE node pools

Configure log export

1. Run the following shell script in the GCP project you've selected for deployment.

Note: Be sure to replace <your-subscription-name> and <your-topic-name> with your own values.

copydownload
wget https://raw.githubusercontent.com/dynatrace-oss/dynatrace-gcp-function/master/scripts/deploy-pubsub.sh
chmod +x deploy-pubsub.sh
./deploy-pubsub.sh --topic-name <your-topic-name> --subscription-name <your-subscription-name>

2. Configure log export to send the desired logs to the GCP Pub/Sub topic created above.

Dynatrace permissions

You need to create a token with a set of permissions.

1. In the Dynatrace menu, go to Access tokens.
2. Select Generate new token.
3. Enter a name for your token.
4. Under Template, select GCP Services Monitoring.
5. Select Generate.
6. Copy the generated token to the clipboard. Store the token in a password manager for future use.

Alternatively, you can create the token and add permissions manually.

Log ingestion

• If you are using Log Monitoring v1, enable the latest version of Dynatrace log monitoring

• Determine where log ingestion will be performed, according to your deployment. This distinction is important when configuring the parameters for this integration.

  • For SaaS deployments: SaaS log ingest, where log ingestion is performed directly through the Cluster API. recommended

  • For Managed deployments: You have two options:

    • Have the installation script create a dedicated ActiveGate for log ingestion - requires you to create a PaaS token

    • Use an existing ActiveGate for log ingestion

Install

Complete the steps below to finish your setup.

1. Download the Helm deployment package in Google Cloud Shell

bashcopydownload
wget -q "https://github.com/dynatrace-oss/dynatrace-gcp-function/releases/latest/download/helm-deployment-package.tar"; tar -xvf helm-deployment-package.tar; chmod +x helm-deployment-package/deploy-helm.sh

2. Configure parameter values

1. The Helm deployment package contains a values.yaml file with the necessary configuration for this deployment. Go to helm-deployment-package/dynatrace-gcp-functionand edit the values.yaml file, setting the required and optional parameter values as follows.

2. Choose which services you want Dynatrace to monitor.

By default, the Dynatrace GCP integration starts monitoring a series of selected services. Uncomment any additional services you want Dynatrace to monitor in the values.yaml file.

Note: For DDU consumption information, see Monitoring consumption.

3. Connect your Kubernetes cluster

To connect to your existing GKE standard cluster or existing GKE Autopilot cluster, run the command below, making sure to replace

• <cluster> with your cluster name
• <region> with the region where your cluster is running
• <project> with the project ID where your cluster is running

shcopydownload
gcloud container clusters get-credentials <cluster> --region <region> --project <project>

For details, see Configuring cluster access for kubectl.

4. Run the deployment script

The deployment script will create an IAM service account with the necessary roles and deploy dynatrace-gcp-function to your GKE cluster.

You have two options:

• Run the deployment script without parameters if you want to use the default values provided (dynatrace-gcp-function-sa for the IAM service account name and dynatrace_function for the IAM role name prefix):

bashcopydownload
cd helm-deployment-package
./deploy-helm.sh

• Run the deployment script with parameters if you want to set your own values (be sure to replace the placeholders with your desired values):

bashcopydownload
cd helm-deployment-package
./deploy-helm.sh [--role-name <role-to-be-created/updated>]

Verify installation

To check whether installation was successful

1. Check if the container is running.

   Note: After the installation, it may take couple of minutes until the container is up and running.

   copydownload
   kubectl -n dynatrace get pods

2. Check the container logs for errors or exceptions. You have two options:

copydownload
kubectl -n dynatrace logs -l app=dynatrace-gcp-function -c dynatrace-gcp-function-metrics
kubectl -n dynatrace logs -l app=dynatrace-gcp-function -c dynatrace-gcp-function-logs

3. Check if dashboards are imported.

   In the Dynatrace menu, go to Dashboards and filter by Tag for Google Cloud. A number of dashboards for Google Cloud Services should be available.

Enable alerting

To activate alerting, you need to enable custom events for alerting in Dynatrace.

To enable custom events

1. In the Dynatrace menu, go to Settings.
2. In Anomaly detection, select Custom events for alerting.
3. Filter for GCP alerts and turn on On/Off for the alerts you want to activate.

View metrics and logs

After deploying the integration, depending on your deployment type, you can:

• See metrics from monitored services: in the Dynatrace menu, go to Metrics and filter by gcp.
• View and analyze GCP logs: in the Dynatrace menu, go to Logs and, to look for GCP logs, filter by cloud.provider: gcp.

View enabled services

The list of currently enabled services can be found in the cluster's ConfigMap named dynatrace-gcp-function-config.

Add or remove services

To add or remove monitored services, follow the steps below.

1. Edit values.yaml by commenting or uncommenting configuration blocks corresponding to specific services. Note: If you already deleted the deployment package and don't have the original values.yaml file anymore, you can use a new one. In this case, the new file will override your previous configuration, so make sure not to accidentally disable monitoring of previously monitored services.
2. Update monitored services by running the script below.

   bashcopydownload
   cd helm-deployment-package
   ./deploy-helm.sh --upgrade-extensions

3. If you removed services from monitoring, find the relevant extensions in your Dynatrace Hub (in the Dynatrace menu, go to Extensions) and delete them to remove service-specific assets (dashboards, alerts, etc).

Example

In the following example

• The gae_instance service is disabled.
• For the gce_instance service, only two feature sets are enabled: default_metrics and istio.

yamlcopydownload
# Google App Engine Instance
    #- service: gae_instance
    #  featureSets:
    #    - default_metrics
    #  vars:
    #    filter_conditions: ""
    # Google VM Instance
    - service: gce_instance
      featureSets:
        - default_metrics
    #    - agent
    #    - firewallinsights
        - istio
    #    - uptime_check
      vars:
        filter_conditions: ""

For a complete list of the GCP supported services, see Google Cloud Platform supported service metrics.

Change deployment settings

• To change the deployment type (all, metrics, or logs), see Change deployment type.
• To change which services are monitored, see Add or remove services.
• To change other settings in values.yaml, see Change parameters from values.yaml.

Change parameters from values.yaml

To load a new values.yaml file, you need to upgrade your Helm release.

To update your Helm release

1. Find out what helm release version you're using.

   copydownload
   helm ls -n dynatrace

2. Run the command below, making sure to replace <your-helm-release> with the value from the previous step.

   copydownload
   helm upgrade <your-helm-release> dynatrace-gcp-function -n dynatrace

For details, see Helm upgrade.

Change deployment type

To change the deployment type (all, metrics, or logs)

1. Find out what Helm release version you're using.

   copydownload
   helm ls -n dynatrace

2. Uninstall the release.

   Note: Be sure to replace <your-helm-release> with the release name from the previous output.

   copydownload
   helm uninstall <your-helm-release> -n dynatrace

3. Edit deploymentType in values.yaml with the new value and save the file.

4. Run the deployment command again. For details, see Run the deployment script.

Troubleshoot

To investigate potential deployment and connectivity issues

1. Verify installation
2. Enable self-monitoring optional
3. Check the dynatrace_gcp_<date_time>.log log file created during the installation process.

• This file will be created each time the installation script runs.
• The debug information won't contain sensitive data such as the Dynatrace access key.

4. If you are contacting Dynatrace ONE:
   • Make sure to provide the dynatrace_gcp_<date_time>.log log file described in the previous step.
   • Provide version information.
     • For issues during installation, check the version.txt file.
     • For issues during runtime, check container logs.

Uninstall

1. Find out what Helm release version you're using.

copydownload
helm ls -n dynatrace

2. Uninstall the release.

Note: Be sure to replace <your-helm-release> with the release name from the previous output.

copydownload
helm uninstall <your-helm-release> -n dynatrace

Alternatively, you can delete the namespace.

bashcopydownload
kubectl delete namespace dynatrace

3. To remove all monitoring assets (dashboards, alerts, etc) from Dynatrace, you need to remove all GCP extensions.

To remove an extension

1. In the Dynatrace menu, go to Extensions and search for the GCP extensions.
2. Select an extension you want to remove, and then select the trash icon in the Actions column to remove it.

Repeat the procedure until you remove all GCP extensions.

Monitoring consumption

Metric ingestion

All cloud services consume DDUs. The amount of DDU consumption per service instance depends on the number of monitored metrics and their dimensions (each metric dimension results in the ingestion of 1 data point; 1 data point consumes 0.001 DDUs). For details, see Extending Dynatrace (Davis data units).

Log ingestion

DDU consumption applies to cloud Log Monitoring. See DDUs for Log Monitoring for details.