Every change in your code can potentially introduce a performance regression and can impact the application’s ability to scale. Regression analysis addresses this problem by tracking performance aspects of different components throughout the development process and under different load patterns.

Black vs. White Box Analysis

There are different flavors of regression analysis. We can either look at the overall response time of a transaction/request or at the execution times of all involved components. The term “Black Box” and “White Box” testing is commonly used where Black Box testing looks at the application as one entity. White Box testing on the other side analyzes the performance of all individual components. At first glance this might not be a big deal as both approaches allow us to identify changes in the application that affect end user response times. But let’s have a look at where White Box testing really rocks!

Simple Scalability Analysis

When you want to test how your application scales I recommend using a simple increasing-workload load test. In this test you start with minimum load and ramp it up over time. Using a Black Box approach we analyze the response time of our application. The following illustration shows how our application behaves from a response time perspective:

Black Box Testing analysis response time. We can probably predict how response time of the app will change with increasing load - or not?
Black Box Testing analysis response time. We can probably predict how response time of the app will change with increasing load – or not?

Looks like our application is doing ok. With increasing load we see response time going up slightly – which can be expected. But – does this mean that this will continue if we add more load on the system? Black Box testing only allows us to assume that the application continues scaling in the same way – but we are not really sure. With White Box Testing we can see into the individual components and can figure out how performance changes in the individual layers.

White Box Testing analysis all involved components. We can see which components scale better than others and how good the application really scales
White Box Testing analyzes all involved components. We can see which components scale better than others and how well the application really scales

Getting “White Box” insight into application components while executing a load test allows us to learn more about the scalability of our application. In this case, it seems like our Business Logic layer is by far the worst scaling component in our application when we increase load. Knowing this allows us to a) focus on this component when improving performance and b) allows us to make a decision on whether and how to distribute our application components.

Analyzing Regressions on Component Level

During the development process you also want to verify if code changes have any negative impact on performance. An improved algorithm in one component can improve overall performance but can also have a negative impact on other components. The improvement effort is neutralized by other components that can’t deal with the changed behavior.

Let’s have a look at an example. We have an application that contains Presentation, Business and Database Layers. An analysis of the first implementation shows that the Business Layer has many roundtrips to the database to retrieve objects that hardly ever change. The engineers decide that these types of objects would be perfect for caching. The additional Cache Layer frees resources on the database that can be used for other applications. In the next iteration the business owner decides to change certain aspects of the business logic. This change doesn’t seem to have any negative impact on the response time as reported from the Black Box Tests. When looking at all involved components, however, we see that the changed Business Logic is bypassing the Cache because it requires other objects that have not been configured for caching. This puts the pressure back on the database and is good example for a component level regression.

Identify Performance Regressions on Component Level
Identify Performance Regressions on Component Level

Where to go from here?

It doesn’t really matter which tools you are using. Whether you use open source or commercial load testing tools or whether you use commercial performance management software like dynaTrace or your own home grown performance logging mechanisms. It is important that you run tests continuously and that you track your performance throughout the development lifecycle. You are more efficient if you have tools that allow you to automate many of these tasks such as automatically executing tests, automatically collecting performance relevant data on code level and also automatically highlighting regressions or scalability problems after tests are executed.

To give you an idea – here are some screenshots of how automated and continuous White Box Testing can help you in your development process.

Track performance of tests across builds and automatically alert on regressions

Automatically Identify Performance Regressions on your daily performance and integration tests
Automatically Identify Performance Regressions on your daily performance and integration tests

Analyze an Increasing Load test and identify the problematic transactions and components

Analyze scalabilty characteristics of components during an increasing load test
Analyze scalability characteristics of components during an increasing load test

Analyze regressions on code level between builds/milestones

Compare performance regressions on component and also method level
Compare performance regressions on component and also method level

Additional Reading Material

If you want to know more about testing check out the other blog posts such as 101 on Load Testing, Load Testing with SilkPerformer or Load Testing with Visual Studio

An improved algorithm in one component can improve overall performance but can also have a negative impact on other components. The improvement effort is neutralized by other components that can’t deal with the changed behavior.

Analyzing Regressions on Component Level

Let’s have a look at an example.We have an application that contains a Presentation, Business and Database Layer. An analysis of the first implementation shows that the Business Layer has many roundtrips to the database to retrieve objects that hardly ever change. The engineers decide that these type of objects would be perfect for caching. The additional Cache Layer frees resources on the database that can be used for other applications. In the next iteration the business owner decides to change certain aspects of the business logic. This change doesn’t seem to have any negative impact on the response time as reported from the Black Box Tests. When looking at all involved components we however see that the changed Business Logic is bypassing the Cache because it requires other objects that have not been configured for caching. This puts the pressure back to the database and is good example for a component level regression.

Identify Performance Regressions on Component Level
Identify Performance Regressions on Component Level