How can I set up outgoing problem notifications using a webhook?

Dynatrace offers several out-of-the-box integrations that automatically push problem notifications to third-party incident-management and ChatOps systems. If however your third-party system isn't supported with an out-of-the-box integration, you can easily set up a customizable webhook integration. Using this approach, whenever Dynatrace detects a problem in your environment that affects real users, a webhook triggers an HTTP POST request to a target URL that you specify. The payload message of the HTTP POST request is completely customizable. By default it has to represents valid JSON syntax except that you define a different HTTP content type header. In case you define a different HTTP content type header Dynatrace will skip the JSON validation. Information placeholders, such as {ProblemTitle} and {State}, are used to fill the custom JSON with the dynamic information of each detected problem.

To integrate problem-notifications using a custom webhook

Navigate to Settings > Integration > Problem notifications.

Click Set up notifications, and select Custom integration, as shown below.

The Set up custom integration page appears.

Type the following information to set up a webhook integration

  • Name User-defined display name that distinguises this unique integration from other integrations.
  • Webhook URL Target URL where the HTTP POST should push the JSON payload. This URL can contain HTTP parameters such as an authentication token, in case the destination system works with authentication tokens instead of basic authentication.
  • Additional HTTP headers (optional) Specify additional HTTP header fields, such as 'Content-Type' or 'Authorization'. These custom HTTP header fields can be used if the target endpoint needs an authentication token within the HTTP header or if you would like to send different content type such as 'text/plain' or 'application/xml'.
  • Create basic authorization header (optional) Basic authorization HTTP headers represents a username password pair that is concatenated by a colon character and finally base64 encoded. Given an example of a username 'alice' that comes with a password 'hello.my.pwd'. Both values are combined with a colon 'alice:hello.my.pwd' and base64 encoded, which finally results in 'YWxpY2U6aGVsbG8ubXkucHdk'. The basic authorization HTTP will therefore look like: 'Authorization' : 'basic YWxpY2U6aGVsbG8ubXkucHdk'. To save you the time to repeat the process for your own username and password, the UI button 'Create basic authorization header' automatically generates that specific HTTP header for you just by entering your username and password.
  • Custom JSON payload
    Once a problem is detected or resolved, this customizable JSON payload is pushed through an HTTP POST to the target system. Use specific placeholders to dynamically populate the JSON payload with problem-related information, such as problem state or title.

Finalize, send a test notification, and click Save.

Use the Notification level switches to control the types of problem notifications that are sent to the target system (problems with Applications, Services, and/or Infrastructure).

Finally, before you save your custom integration setup, click Send test notification to perform a test run and ensure that the destination address is available.

Example

Here is a valid JSON web hook problem-notification definition example.

{
   "State" : "{State}",
   "ProblemID" : "{ProblemID}",
   "ProblemTitle" : "{ProblemTitle}"
}

Once a problem is detected, the placeholders are populated with the actual values and results, as shown in this example payload:

{
   "State" : "OPEN",
   "ProblemID" : "512",
   "ProblemTitle" : "OPEN Problem 912: Elastic load balancer has a high backend failure rate on Elastic Load Balancer help-LB"
}

Available placeholders

  • {State}: Problem state. Possible values are OPEN and RESOLVED.
  • {ProblemID}: ID number of the reported problem.
  • {PID}: Unique system identifier of the reported problem.
  • {ProblemImpact}: Impact level of the problem. Possible values are APPLICATION, SERVICE, and INFRASTRUCTURE.
  • {ProblemTitle}: Short description of the problem.
  • {ProblemDetails}: All problem event details, including root cause, as an HTML-formatted string.
  • {ProblemURL}: URL of the details page of the corresponding problem in Dynatrace.
  • {ImpactedEntity}: Entities impacted by the problem (or the term "multiple" when more than two entities are impacted).
  • {ImpactedEntities}: A JSON array of all impacted entities in the form of:
[
   { "type" : "HOST", "name" : "MyHost1", "entity" : "HOST-XXXXXXXXXXXXX" },
   { "type" : "SERVICE", "name" : "MyService1", "entity" : "SERVICE-XXXXXXXXXXXXX" }
]
  • {Tags}: Comma-separated values list of tags that have been defined for all impacted entities.