How can I filter problem notifications with alerting profiles?

Alerting profiles enable you to set up fine-grained alert-filtering rules that are based on the severity, customer impact, associated tags, and/or duration of detected problems. Alerting profiles allow you to control exactly which conditions result in problem notifications and which don’t. This includes all problem-push notifications that are sent via the Dynatrace mobile app and displayed in the Dynatrace web UI. Alerting profiles can also be used to set up filtered problem-notification integrations with 3rd party messaging systems like Slack, HipChat, and PagerDuty.

How alerting profiles work

Alerting profiles control the delivery of problem notifications across your organization’s alerting channels by considering predefined filters that are based on problem severity, problem duration, and tags. Alerting profiles control all alerts that appear in the Dynatrace UI, alerts that are sent via the Dynatrace mobile app, and the alerts that are sent via 3rd party integrations, such as Slack and HipChat.

Each of your monitoring environments has a default alerting profile that defines the level of severity that must be met before an alert is sent out for a detected problem. The default alerting profile applies to all users within your environment—if/until your environment admin creates a new alerting profile and assigns it to individual users.

Your environment administrator can create new alerting profiles that provide fine-grained control over the routing of problem alerts for individual teams or for 3rd party problem notification integrations, such as Slack, HipChat, etc.

To view your environment’s alerting profiles, go to Settings > Alerting > Alerting profiles. The list of alerting profiles in the example below includes the Default profile and two customer-defined team-specific profiles (Team Detroit and Team Linz).

Note: The default profile can be modified but not deleted.

Create an alerting profile

Alerting profiles provide a powerful filtering mechanism for problem notifications. By combining filter criteria, you can create custom profiles that, for example, delay notification of problems in development environments while immediately alerting on problems detected in production environments.

To create an alerting profile

  1. Go to Settings > Alerting > Alerting profiles.

  2. Type a name for the new profile in the Create new alerting profile field.

  3. Click the Create button.

  4. On the Edit profile page, click the Create alerting rule button to add a rule to the new profile.

  5. Select a severity level from the Problem severity level drop list. Severity options include Availability, Error, Performance, or Resource.

  6. Type in a number of minutes to define how long a problem duration should last before an alert is sent out. This enables you to avoid alerts for fleeting, low-severity problems that don’t affect customer experience and therefore don’t require immediate attention.

  7. You can further filter problems down to specific monitored entities in your environment by selecting an existing tag from the Select tag field. The Filter problems by tag drop list enables you to define a logical operator for the tags you select. Only include entities that have all tags (i.e., AND) enables you to create combined tag rules. Include entities that have at least one tag (i.e., OR) enables you to create either/or rules for tags.

Alerting profile example

Say you want to receive immediate alerts on availability problems that involve production entities, but you only want to receive alerts on development availability problems that are open longer than 15 minutes. To model this example, you would create two availability filtering rules in your alerting profile, as shown below.

Once you’ve created a new alerting profile, you need to assign the profile to specific users. The assigned profile will control the problem notifications that are sent to these users and the problem alert in the upper right-hand corner of the top bar (see image above).

Third-party notification integrations

All third-party problem notification integrations now support alerting profiles. The seamless use of alerting profiles for third-party problem integrations allows efficient filtering regardless of the alerting channel your team uses (web UI, Dynatrace mobile app, Slack, HipChat, ServiceNow, etc).

For example, to check this for Slack

  1. Go to Settings > Integration > Problem notifications.
  2. Click Set up notifications.
  3. Select Slack.
  4. You can find on this page, an Alerting profile list box (see image below).

Migrate from watch settings to alerting profiles

If you're still using Dynatrace problem watch settings, which have been deprecated, when you enable alerting profiles, any existing watch settings that you've configured will be overwritten by the rules in your active alerting profiles—you will no longer be able to view or access your existing watch settings. Additionally, any tag-based filters or configurations you have set up for 3rd party messaging integrations will also be overwritten by your alerting profiles. Until you set up custom alerting profiles for specific integrations and users, notifications will be sent out based on the filtering rules set up in the default alerting profile.

Existing personalized watch settings can’t be migrated automatically to alerting profiles, you must manually migrate your watch settings. Note that newly created Dynatrace environments don’t require this migration step and alerting profiles are enabled by default.

To enable alerting profiles in an existing environment

  1. Have your environment administrator select Problems from the navigation menu.

  2. Click the Enable alerting profiles button (see below).