Regression analysis

What is regression analysis?

Regression analysis estimates the differences between application executions. If you fix bugs, introduce new features or refactor an application, you affect other application areas. Regression analysis estimates how your application performs if you:

  • Have two different application versions or releases.
  • Execute the application on two different environments or platforms.
  • Change third-party libraries.

Goal of this tutorial

To describe how to do a simple regression analysis based on a problem pattern in easyTravel.

Create a dashboard to pinpoint the differences in two recorded sessions. Use one problem pattern and show the difference.

Scenario

Assume that you have recently refactored your application and modified some APIs. Also, you fixed some bugs and created new features. The main features remain the same. Determine if the changes affect the main features negatively or positively. Use the SlowAuthentication problem pattern in easyTravel to demonstrate the two different application variants. Do not deploy two different versions of easyTravel.

Requirements

Install, configure and run the following components:

  1. Click easyTravel Configuration UI > Install System Profile.
  2. Click UEM - Lightweight to start a scenario.
  3. Select Manual Visits.
  4. Click Cockpits > Status Overview > Agent Overview to make sure that all Agents are connected to the AppMon Server.
  5. Click Cockpits > System Profiles > easyTravel > Diagnose Performance > PurePaths to make sure you get events and PurePaths in your live session.

Detailed steps

The following is a high level overview of the testing process:

  1. Create a repeatable automated easyTravel test suite (with Selenium) or use the basic example Selenium test suite for this and the next step.
  2. Automate a session recording through the AppMon REST interface.
  3. Create a Regression Analysis dashboard to show the performance differences.

If you already have two comparable stored sessions, you can skip to Step three.

Use an easyTravel problem pattern to simulate a regression.

Step One: Create Repeatable Automated Tests

For a meaningful regression test, make sure you compare the same scenarios or identical test cases.

For web applications, use Selenium to create an automated test suite that you can replay. Install Selenium as a Mozilla Firefox plugin and click on various functions in your application to record test cases. You can use any tool that is capable of test-driving a web app.

For this scenario, use the easyTravel customer frontend login and the SlowAuthentication problem pattern to simulate two different easyTravel versions:

  1. Start the easyTravel customer frontend.

  2. Click the Lock icon to get a list of predefined users.

  3. Click monica.. Record this in Selenium or look at the edited result in its IDE:

  4. Open Test Suite > Test Suite Part 1.

  5. Double-click Login monica Test Case.

Step Two: Automate Session Recording with AppMon's REST Interface

You must compare two identical sessions. Use AppMon's REST interface to automate the session recording.

The following are the necessary functions:

Perform these on the easyTravel System Profile. You can embed the necessary commands to operate AppMon in the Selenium test suite. Follow these steps:

  1. Clear the AppMon live session.
  2. Start the recording session.
  3. Execute test suite steps.
  4. Stop recording.

Follow these steps to complete this step:

  1. Click easyTravel Configuration UI > Install System Profile.
  2. Click UEM - Lightweight to start a scenario.
  3. Select Manual Visits.
  4. Execute the (Selenium) test suite.
  5. The problem patterns are hidden by default. To show them, click the Settings icon in the easyTravel Configuration UI.

Step Three: Create a Regression Analysis Dashboard

Check the recorded session

Two stored sessions appear in the AppMon Server if the automated test suite executes successfully. The sessions should be identical because there are no other requests running while you record. Take a look at the sessions and verify that:

  • All PurePaths are complete.
  • You have the same amount of PurePaths in the sessions.
  • The PurePaths have about the same length (same instrumentation depth).

In a real-world scenario, the sessions are different and you may have a different number of PurePaths.

Comparison dashboard

  1. Create a new empty dashboard (^N).
  2. Right-click the Comparison tab.
  3. Click the Properties dialog box.
  4. Set the data source to the stored session 1, which is the baseline for the regression.
  5. Right-click the Source tab.
  6. Click the Properties dialog box.
  7. Set the data source to the stored session #2. Do not apply filters.

Drag the following dashlets to the dashboard so they appear on the same page:

  • API Breakdown Dashlet
  • Database
  • Web services
  • Remoting
  • Method
Example Regression Dashboard
Example Regression Dashboard

Due to the SlowAuthentication artificial regression, some timings differ drastically. You may want to explore this deeper.

Conclusion

AppMon provides regression information and quickly puts you on the path to discovery, even if you are not familiar with the application. For best results, use the AppMon automation interface to perform regression analysis in an automated environment.