Overall performance metrics - test analysis

The Overall Performance Metrics section of the Test Analysis page compares the performance metrics for this test as seen over the last hour against the baseline performance metrics as seen over the last 24 hours. If the current performance is consistent with the baseline performance, the test passes. Otherwise the test fails and you may want to investigate.

The thresholds for the pass/fail criteria within this section are:

  • End-to-end time – If the end to end times for this test during the last hour fall above their historical thresholds plus three deviations above the geometric mean for the last 24 hours, this test will fail.
  • DNS time – If the DNS times for this test during the last hour fall above their historical thresholds plus three deviations above the geometric mean and are above 90 milliseconds, this test will fail.
  • Connection time – If the connection times for this test during the last hour fall above their historical thresholds plus three deviations above the geometric mean and are above 150 milliseconds, this test will fail.
  • SSL time (if applicable) – If the SSL times for this test during the last hour fall above their historical thresholds plus three deviations above the geometric mean and are above 100 milliseconds, this test will fail.

Geometric mean

The test analysis relies on the Geometric Mean to reduce the effect of outlier and to determine whether the end-to-end, DNS, and connection time are within normal ranges for an application.

This analysis mainly uses arithmetic averaging for aggregating and displaying data. This method takes the sum of a set of numbers, and divides it by how many numbers there are in the set. This method is very inclusive of the effect of outlying points.

Because analysis moves towards analyzing data to make decisions on which components of an application are behaving outside their normal bounds, it relies upon geometric averaging to reduce the impact of outliers.

The geometric mean has the effect of minimizing the skewing that can be seen by outlying values, leading to an average that is more reliable from a pattern recognition perspective.

The following is an example of the difference between arithmetic and geometric averages on the same data set:

Sample data set:

  • Measurement 1 – 1.0
  • Measurement 2 – 3.0
  • Measurement 3 – 4.0
  • Measurement 4 – 4.0
  • Measurement 5 – 5.0
  • Measurement 6 – 12.0

Arithmetic mean:

=Sum of data points/# of data points
=(1+3+4+4+5+12)/6
=29/6
=4.8333...

Geometric mean:

=10^(Sum of the log10 of each data point/# of data points)
=10^(log10(1)+log10(3)+log10(4)+log10(4)+log10(5)+log10(12)/6)
=10^(0+0.477+0.602+0.602+0.699+1.079/6)
=10^(3.457/6)
=10^(0.576)
=3.767