Integrate PagerDuty with Dynatrace Cloud Automation
Integrating Dynatrace Cloud Automation with PagerDuty empowers teams to take the right action and resolve incidents faster, mobilizing responders whenever significant problems occur in Dynatrace while coordinating other actions in Cloud Automation.
To integrate Dynatrace Cloud Automation with PagerDuty, follow the instructions below.
Prerequisites
PagerDuty
Dynatrace
PagerDuty
-
You need a license that includes Event Routing Rules and Alert Field Enrichment and Extraction (typically Business or Digital Operations).
-
Add an Incoming Event Source for Cloud Automation.
-
Add a PagerDuty REST API key that will be used to configure the webhook translator in AWS Lambda.
-
Set up a Cloud Automation webhook translator to receive PagerDuty events from the PagerDuty event ruleset and make the Cloud Automation API call to send back a Cloud Automation sequence task finished event.
Dynatrace
-
Create a problem notification webhook.
-
Create a Cloud Automation project with a shipyard file that has a custom task (for example,
openticket
) to trigger the PagerDuty subscription.
Set up the PagerDuty webhook subscription
Use case: The Cloud Automation sequence will wait for PagerDuty Cloud Automation webhook translator to send back a Cloud Automation sequence task finished event.
Cloud Automation
PagerDuty
Cloud Automation
-
On the Cloud Automation bridge, select your Cloud Automation project.
-
Select Settings, and then select Integrations.
-
Select webhook-service from the list, and then select Add subscription.
-
Enter the following values:
- Task: Select the task associated with your shipyard file (in this case,
openticket
). - Task suffix: Select
triggered
. - Request method: Select
POST
. - URL: Enter your API endpoint using the integration key obtained in Prerequisites (for example,
https://events.pagerduty.com/x-ere/<YOUR-INTEGRATION-KEY?
).
- Task: Select the task associated with your shipyard file (in this case,
-
Select Add custom header and enter the following values:
- Name: Enter
Content-Type
. - Value: Enter
application/json
.
- Name: Enter
-
Enter a Custom payload.
Example:
{ "data": { "project": "{{.data.project}}", "service": "{{.data.service}}", "stage": "{{.data.stage}}" }, "shkeptncontext": "{{.shkeptncontext}}", "id": "{{.id}}", "type": "{{.type}}", "incident": { "ProblemID": "{{.data.incident.ProblemID}}", "ProblemTitle": "{{.data.incident.ProblemTitle}}", "ProblemURL":"https://<YOUR DYNATRACE BASE URL>/ui/problems?displayId={ProblemID}", "State": "{{.data.incident.State}}", "Tags": "{{.data.incident.Tags}}" } }
-
Under Send started event, select
automatically
. -
Under Send finished event, select
by webhook receiver
. -
Select Create subscription.
PagerDuty
Below is a ruleset example. You can adjust the PagerDuty Event Rules to route to different services depending on the application, project, stage, and so on.
-
In PagerDuty, select Automation > Event rules, then select the ruleset whose integration key you used in Cloud Automation.
-
Select New Event Rule. You should be able to see the test event you sent to PagerDuty. Select the
+
button to expand it. -
Choose to run this rule when the following condition is met:
type
equalssh.keptn.event.openticket.triggered
. -
Expand Customize event fields, then under Define custom variable, add the following variables:
Name Regex Source shkeptncontext
.*
shkeptncontext triggeredid
.*
triggeredid project
.*
data.project service
.*
data.service stage
.*
data.stage ProblemID
.*
incident.ProblemID ProblemTitle
.*
incident.ProblemTitle ProblemURL
.*
incident.ProblemURL Tags
.*
incident.Tags -
Under Replace event field, add the following fields:
Event Field Value Template/Regex dedup_key
{{shkeptncontext}} - {{triggeredid}}
Template summary
{{ProblemID}} - {{ProblemTitle}}
Template class
{{stage}}
Template component
{{service}}
Template source
{{project}}
Template group
openticket Template custom_details > shkeptncontext
{{shkeptncontext}}
Template custom_details > triggeredid
{{triggeredid}}
Template custom_details > ProblemURL
{{ProblemURL}}
Template custom_details > Tags
{{Tags}}
-
Expand Do these things, then select Create an incident on a service and select an appropriate PagerDuty service.
-
Select Advanced and enter the following values:
-
Custom Action: Enable
- Enable Custom Action
- Trigger automatically on incident creation
-
Name: Enter, for example,
Send finished event to Dynatrace Cloud Automation
. -
API endpoint Enter the URL of the API gateway trigger that you configured for your translator Lambda in AWS.
-
-
Select Save rule.
Trigger a sequence
Cloud Automation triggers PagerDuty whenever a Dynatrace problem is detected, and the Dynatrace problem notification event is sent to Cloud Automation.
To trigger the sequence, follow the steps below.
-
Trigger a problem in Dynatrace.
-
In PagerDuty, go to Incidents > Alerts and search for an alert named
Incident routed via a Ruleset
that occurred around the time when you triggered the problem. -
Select the alert, then select View message. Similar content should be displayed:
{ "event": "sh.keptn.event.evaluation.finished", "incident_key": "7ee74459136b4b7ca694efa7b6164764", "project": "demo", "result": "fail", "score": 61.11111111111111, "service": "casdemoapp", "stage": "production" }
If no PagerDuty incident is created, review the configuration steps and make sure you entered all information correctly.
-
On the Cloud Automation bridge, go to Sequences and check if the
incident
sequence was executed. -
Select the incident sequence, then select the
openticket
task in the sequence.Expected results:
webhook-service
should have started- The
pagerduty
task should have finished successfully (pass
)