Types of alerts

The following table shows which alerts can be configured for the different test types.

Alert Type Backbone Mobile Private Last Mile
Response time – test X X X
Response time – step X
Transaction Failure – test X X X
Transaction Failure – step X
Object Failure X
Byte Limit X

Aligning alert types with performance goals

The different alert types are useful for different goals. For example:

  • To ensure that SLAs are being met, use Transaction Failure alerts to monitor availability and static Response Time alerts to monitor page speed.
  • To monitor for changes in third-party content that may affect page loads, use Byte Limit alerts.
  • To know when content is missing from a page even though the overall page load is successful, use Object Failure alerts.
  • To monitor for changes in system behavior so you can diagnose and correct underlying causes, use dynamic Response Time alerts.
  • When local conditions are a factor in performance, so different monitoring locations need different thresholds relative to their normal performance, use dynamic Response Time alerts.

Alert types

Response Time alerts

A Response Time alert is triggered when the response time is longer than the threshold you set.

Depending on your testing scenario, you can configure either a dynamic threshold or a static threshold for a Response Time alert. Mobile tests have a third configuration option, average threshold. For details, see Alert thresholds.

Use case

You want to make sure you're meeting response time SLAs for transactions and for each page in a transaction.  You have run tests to determine the baseline response times, and have calculated the thresholds for triggering when that time is exceeded.

Creating Response Time alerts for a test, with dynamic or static thresholds, provides the flexibility to define the thresholds as a percentage or in seconds (determined by the standard deviation for the test runs), to trigger the alerts based on how many times I’ve exceeded it in a given period.

Transaction Failure alerts

A Transaction Failure alert is triggered when a predetermined number or percentage of nodes (Backbone tests), locations (Mobile), or peers (Private Last Mile tests) fail to properly complete a transaction. Transactions can fail for many reasons, such as:

  • Failure to reach the designated host
  • Content match failures
  • No successful objects returned for a page
  • Timeouts exceeded prior to downloading all objects for a test
  • Global portal byte limits for the test are exceeded
  • User script failures

When you configure Transaction Failure alerts, it is particularly important to include content validation in the transaction, to rule out script errors as the cause of the failure.

We recommend configuring the alert to be triggered after a minimum of two failures on multiple nodes.

Retry on error for Backbone tests

In Backbone tests, enable Retry on Error to avoid "false alarms". If a test fails because of certain issues outside the test or your website, the test is automatically run again. If the second run is successful, the failed first test run is discarded. Best practice is to enable Retry on Error for all Backbone tests. For more information, see Backbone alert settings.

Use case

To make sure you're meeting SLAs for availability, you want to be notified when a transaction does not complete successfully. You need flexibility in defining a transaction: a test consisting of multiple steps, a step consisting of one or more pages, or a single page. You also need the flexibility to define how many consecutive failed transactions trigger an alert notification.

Object Failure alerts

An Object Failure alert is triggered when the specified number or percentage of objects for an entire transaction fail to load.

If the base HTML of a page loads successfully, the test run is reported as successful even if important content hosted by other servers or networks does not load successfully. You need Object Failure alerts to ensure that third-party content is displayed correctly.

The page object status for a given test at a given node is calculated by evaluating the number or percentage of object errors for each test compared to the total number of requested objects.

Byte Limit alerts

A Byte Limit alert is generated when the total number of bytes downloaded during a successful test execution exceeds the specified byte size limit.

The Byte Limit alerts notify you when there are significant changes to the size of the pages returned to the portal. Changes to a page size can signify that the content is not being delivered properly or that a code change has had an adverse impact on the end users' experience.

Use case

You want to know when the size of objects downloaded during a transaction exceeds a certain number of bytes. You prefer to keep the pages in a standard configuration and submit changes through a change review process. You need to know both the impact of planned changes and whether third party services have made changes to the page content.

You know the baseline byte count of the transaction; an increase of more than X percent, indicates that your organization, or a third party, is inserting new code, ads, or other content in your web pages, often without the knowledge of the application owners.

By monitoring the byte count, you can determine the potential impact of page objects on network utilization and application performance. You can also detect that new objects are added to the page, in particular when a third party service changes the page configuration or payload.

Step level alerts

For Backbone tests, you can configure Step Level alerts for Response Time and Transaction Failure. Step Level alerts have the same configuration details as Test Level alerts, and the same planning and configuration guidelines apply.

Use Step Level alerts for critical business processes within a transaction, for example logging in or submitting payments. Step Level alerts are also useful when a step's performance is a better problem indicator than the overall transaction; for example, one page that loads slowly affects end users' experience even though the overall transaction has a normal total response time.

Alert limitations

  • You cannot use alerts to monitor the performance of individual objects such as images or third-party tags. Object Level alerts respond to the aggregate performance of objects in the transaction, not to deviations in specific individual objects.
  • A new alert won't be triggered if the test status changes from Severe to Warning. Although that's a performance improvement, you may need to know that performance has not yet returned to normal. Configuring alerts to send reminders will guarantee that notifications will be sent in this situation.