Analyzing test automation data

Refer to the options below to analyze test automation data in AppMon or in your continuous integration server environment.

Test Overview dashlet

The Test Overview dashlet displays the general status of tests in your environment.

Test Overview dashlet
Test Overview dashlet

Note

This dashlet is only available in the Pre-Production (formerly Test Center) Edition.

Bar chart section

In the upper section of the dashlet, a bar chart displays information about the overall number of test executions with each status. The X-axis contains the aggregation specific keys and the Y-axis shows the number of matching test executions. You can choose the aggregation type in the dashlet header.

Aggregation types

The following aggregation types are available:

  • By build: Test executions with the same build version parameters (major.minor.revision.build) are aggregated.
  • By day: Test executions done in the same day are aggregated.

Pie charts section

Below the Bar Chart section, pie charts show the number of test executions with each status, as well as the specified test category such as Unit Tests, Performance Tests, Web API Tests, Browser Tests. There is one pie chart per each test category. Only pie charts with a test category that have at least one test execution are shown. As a result, one to four pie charts can be visible in this section.

By default, the pie charts filter test executions by matching test category and show a consolidated view of the details for all your builds for the selected timeframe.

Selecting any bar in the Bar Chart section by clicking it once also filters the data shown in Pie Charts section by aggregation specific key, as shown in the bar's X-axis. Clear the selection by clicking the chart above the bar or between the bars. The current selection has a thin border around the bar and the header of the Pie Charts section.

Drilldowns to Test Results dashlet

You can drill down from the Test Results Dashlet dashlet to see more detailed, filtered data by doing one of the following:

  • Double click on any bar in the Bar Chart section. This filters data by an aggregation specific key, as shown in the bar's X-axis.
  • Single click on the chart or chart title in Pie Charts section. This filters the data by a related test category to the chart and the aggregation specific key, as shown in the bar's X-axis if any bar is selected. You can also check it on the header of Pie Charts section.

A Test Results dashlet opened by drilldown inherits all filters applied to the source Test Overview dashlet. For example, the selected timeframe.

Load tests results

Load tests can be analyzed using a separate Load Test Overview dashboard. You can open it by clicking the Analyze Load Test Results link in Test Overview dashlet footer. See Load Tests for more information.

The Footer section also includes useful documentation links about tests.

Test Results dashlet

The Test Results dashlet visualizes the results of various tests in your environment.

Test Results dashlet
Test Results dashlet

This dashlet is only available in the AppMon Test Center edition.

Test case section

The upper section of the dashlet lists the test cases, organized by defined test category and package. Each test case entry shows an aggregated view of the KPIs.

Test status

Every test case has a status:

  •  OK: Test case executed correctly.
  •  Degrading: Test case runs have become slower.
  •  Improving: Test case runs have become faster.
  •  Volatile: Test case has a very volatile outcome — sometimes faster, sometimes slower — so changes in performance might not be recognized correctly. This especially happens with very short-running tests. Try to increase the duration of the test, perhaps by executing the same operation multiple times.
  •  Failing: Test case has a functional problem, so no performance data could be recorded.
  •  Invalidated: The last test run of this test case was manually invalidated by the user.
    When you click a test status icon in the toolbar in the upper right corner of the Test Results dashlet, related test status columns display in the measure section. For example, click the Degraded icon to include the Degraded Runs column or the Volatile icon to include the Volatility column.

Grouping by package can be toggled by clicking the Group by packages icon.

Viewing details

Right-click the item and select Details to view details for a test case, measure, or test run. You can copy information from the Details dialog box.
Details dialog box

Measures section

The measure section in the lower left of the dashlet shows the latest values of the KPIs associated with a test case, and indicates whether those values are within, above, or below the corridor.

Chart section

The chart section shows the historical values of a KPI for a test case, including the calculated performance corridor.

Viewing markers

Any marker set in the test metadata displays in the chart's heat field.

See the Test Automation FAQ for more information.

Load Test Overview dashboard

When you use the Load Testing Integration to run load tests, you can analyze the results that start in the Tagged Web Requests dashlet or use the Load Test Overview dashboard.

Launch the dashboard from the Start Center.

Select a stored session or System Profile. Typically, you have a stored session for a previously run load test. The new dashboard that opens up prompts you for the data source. Select a System Profile if you want to analyze the live session or select a stored session.

Analyze the load test result.
The top-left of the dashboard shows the Tagged Web Requests dashlet. All other dashlets on that dashboard are linked with that dashlet. You must select a test name (you can also multi-select). Data from the name or names you select displays in the charts and the dashlets such as Transaction Flow Dashlet, Database Dashlet, and Response Time Hotspots Dashlet. Use this to analyze the root cause of tests that run slow.

Jenkins plugin

The AppMon Jenkins plugin retrieves test automation data from the AppMon Server and displays it through charts and tables on the project and build level. See Test Automation Plugin for Jenkins for installation, configuration, and usage instructions.
AppMon 6.3 and later includes a default setting to accept authentication data only from HTTPS. See Breaking Changes and Potential Pitfalls if you need more information on this.