Integrate PagerDuty with Dynatrace Cloud Automation

Integrating Dynatrace Cloud Automation with PagerDuty empowers teams to take the right action and resolve incidents faster, mobilizing responders whenever significant problems occur in Dynatrace while coordinating other actions in Cloud Automation.

To integrate Dynatrace Cloud Automation with PagerDuty, follow the instructions below.

Prerequisites

PagerDuty

Dynatrace

PagerDuty

• Create a PagerDuty account.

  Note: You need a license that includes Event Routing Rules and Alert Field Enrichment and Extraction (typically Business or Digital Operations).

• Add an Incoming Event Source for Cloud Automation.

  How to add the incoming event source

  1. Log in to your PagerDuty account and go to Automation > Event rules.

  2. Select Create ruleset and enter the following values:

     • Title: Provide a name, such as dynatrace-cloudautomation
     • Team: Assign a team or leave blank to keep the event as a global ruleset.

  3. Select Create ruleset.

• Add a PagerDuty REST API key that will be used to configure the webhook translator in AWS Lambda.

  How to add the REST API key

  1. In PagerDuty, go to Integrations > API access key.
  2. Select Create new API key.
  3. Enter a description for the API key and tick Read-only API key.
  4. Select Create key.
  5. On the Ruleset page, copy and save the Integration key. This value will be used later on to create a CA secret.

• Set up a Cloud Automation webhook translator to receive PagerDuty events from the PagerDuty event ruleset and make the Cloud Automation API call to send back a Cloud Automation sequence task finished event.

  How to create, upload, and configure the translator

  To create the translator:

  1. Log in to your AWS console and navigate to AWS Lambda.
  2. Select Create function, then Author from scratch.
  3. For Name, enter a function name, such as pd-to-cloudautomation.
  4. For Runtime, select Node.js 14.x.
  5. Select Create function.
  6. Select Add trigger.
  7. Select API Gateway, then Create API > HTTP API > Security > Open.
  8. Select Add to add the trigger.

  After the trigger is created, you can upload it and use the URL to configure the Event Rule in PagerDuty.

  To upload the translator:

  1. Download the pre-build Lambda code ZIP file.
  2. In AWS Lambda, select Code, then upload and save the ZIP file.

  To configure the translator:

  1. In AWS Lambda, select Configuration > Environment variables > Edit.

  2. Add the following variables:

     Variable name	Value
     KEPTN_BASE_URL	The base URL from your Cloud Automation environment (for example, https://abcd.cloudautomation.live.dynatrace.com)
     KEPTN_API_TOKEN	Your Cloud Automation API token
     PD_API_ACCESS_KEY	Your PagerDuty API access key

  3. Select Save.

Dynatrace

• Create a problem notification webhook.

  How to create the problem notification webhook

  1. In the Dynatrace menu, go to Settings > Integrations > Problem notifications.

  2. Select Add notification and enter the following values:

     • Notification type: Select Custom integration
     • Display name: Enter Dynatrace Cloud Automation
     • Webhook URL: Enter your URL endpoint, for example [YOUR CLOUD AUTOMATION BASE URL]/api/v1/event

  3. In Additional HTTP headers, you need to add two headers. Select Add item and enter the following values:

     • First header:

       • Name: Enter x-token
       • Value: Enter your Cloud Automation API token from the Cloud Automation bridge
       • Secret header value: Enabled

     • Second header:

       • Name: Enter Content-Type
       • Value: Enter application/cloudevents+json
       • Secret header value: Disabled

  4. Enter a Custom payload.

     Example:

     jsoncopydownload
     {
        "specversion":"1.0",
        "source":"dynatrace notifications webhook",
        "contenttype":"application/json",
        "type": "sh.keptn.event.production.incident.triggered",
        "data": {
           "project":"incident-demo",
           "stage":"production",
           "service":"casdemoapp",
           "dynatraceProblem": {
              "State":"{State}",
              "ProblemID":"{ProblemID}",
              "PID":"{PID}",
              "ProblemTitle":"{ProblemTitle}",
              "ProblemURL":"{ProblemURL}",
              "ProblemDetails":{ProblemDetailsJSON},
              "Tags":"{Tags}",
              "ImpactedEntities":{ImpactedEntities},
              "ImpactedEntity":"{ImpactedEntity}"
           }
        }
     }

  5. Select Save changes.

• Create a Cloud Automation project with a shipyard file that has a custom task (for example, openticket) to trigger the PagerDuty subscription.

Set up the PagerDuty webhook subscription

Use case: The Cloud Automation sequence will wait for PagerDuty Cloud Automation webhook translator to send back a Cloud Automation sequence task finished event.

Cloud Automation

PagerDuty

Cloud Automation

1. On the Cloud Automation bridge, select your Cloud Automation project.

2. Select Settings, and then select Integrations.

3. Select webhook-service from the list, and then select Add subscription.

4. Enter the following values:

   • Task: Select the task associated with your shipyard file (in this case, openticket).
   • Task suffix: Select triggered.
   • Request method: Select POST.
   • URL: Enter your API endpoint using the integration key obtained in Prerequisites (for example, https://events.pagerduty.com/x-ere/<YOUR-INTEGRATION-KEY?).

5. Select Add custom header and enter the following values:

   • Name: Enter Content-Type.
   • Value: Enter application/json.

6. Enter a Custom payload.

   Example:

   jsoncopydownload
   {
    "data": {
     "project": "{{.data.project}}",
     "service": "{{.data.service}}",
     "stage": "{{.data.stage}}"
   },
   "shkeptncontext": "{{.shkeptncontext}}",
   "id": "{{.id}}",
   "type": "{{.type}}",
   "incident": {
     "ProblemID": "{{.data.incident.ProblemID}}",
     "ProblemTitle": "{{.data.incident.ProblemTitle}}",
      "ProblemURL":"https://<YOUR DYNATRACE BASE URL>/ui/problems?displayId={ProblemID}",
     "State": "{{.data.incident.State}}",
     "Tags": "{{.data.incident.Tags}}"
    }
   }

7. Under Send started event, select automatically.

8. Under Send finished event, select by webhook receiver.

9. Select Create subscription.

PagerDuty

Note: Below is a ruleset example. You can adjust the PagerDuty Event Rules to route to different services depending on the application, project, stage, and so on.

1. In PagerDuty, select Automation > Event rules, then select the ruleset whose integration key you used in Cloud Automation.

2. Select New Event Rule. You should be able to see the test event you sent to PagerDuty. Select the + button to expand it.

3. Choose to run this rule when the following condition is met: type equals sh.keptn.event.openticket.triggered.

4. Expand Customize event fields, then under Define custom variable, add the following variables:

   Name	Regex	Source
   shkeptncontext	.*	shkeptncontext
   triggeredid	.*	triggeredid
   project	.*	data.project
   service	.*	data.service
   stage	.*	data.stage
   ProblemID	.*	incident.ProblemID
   ProblemTitle	.*	incident.ProblemTitle
   ProblemURL	.*	incident.ProblemURL
   Tags	.*	incident.Tags

5. Under Replace event field, add the following fields:

   Event Field	Value	Template/Regex
   dedup_key	{{shkeptncontext}} - {{triggeredid}}	Template
   summary	{{ProblemID}} - {{ProblemTitle}}	Template
   class	{{stage}}	Template
   component	{{service}}	Template
   source	{{project}}	Template
   group	openticket	Template
   custom_details > shkeptncontext	{{shkeptncontext}}	Template
   custom_details > triggeredid	{{triggeredid}}	Template
   custom_details > ProblemURL	{{ProblemURL}}	Template
   custom_details > Tags	{{Tags}}

6. Expand Do these things, then select Create an incident on a service and select an appropriate PagerDuty service.

7. Select Advanced and enter the following values:

   • Custom Action: Enable

     • Enable Custom Action
     • Trigger automatically on incident creation

   • Name: Enter, for example, Send finished event to Dynatrace Cloud Automation.

   • API endpoint Enter the URL of the API gateway trigger that you configured for your translator Lambda in AWS.

8. Select Save rule.

Trigger a sequence

Cloud Automation triggers PagerDuty whenever a Dynatrace problem is detected, and the Dynatrace problem notification event is sent to Cloud Automation.

To trigger the sequence, follow the steps below.

1. Trigger a problem in Dynatrace.

2. In PagerDuty, go to Incidents > Alerts and search for an alert named Incident routed via a Ruleset that occurred around the time when you triggered the problem.

3. Select the alert, then select View message. Similar content should be displayed:

   jsoncopydownload
   {
      "event": "sh.keptn.event.evaluation.finished",
      "incident_key": "7ee74459136b4b7ca694efa7b6164764",
      "project": "demo",
      "result": "fail",
      "score": 61.11111111111111,
      "service": "casdemoapp",
      "stage": "production"
   
   }

   Note: If no PagerDuty incident is created, review the configuration steps and make sure you entered all information correctly.

4. On the Cloud Automation bridge, go to Sequences and check if the incident sequence was executed.

5. Select the incident sequence, then select the openticket task in the sequence.

   Expected results:

   • webhook-service should have started
   • The pagerduty task should have finished successfully (pass)