Every change in your code can potentially introduce a performance regression and can impact the application’s ability to scale. Regression analysis addresses this problem by tracking performance aspects of different components throughout the development process and under different load patterns.
Black vs. White Box Analysis
There are different flavors of regression analysis. We can either look at the overall response time of a transaction/request or at the execution times of all involved components. The term “Black Box” and “White Box” testing is commonly used where Black Box testing looks at the application as one entity. White Box testing on the other side analyzes the performance of all individual components. At first glance this might not be a big deal as both approaches allow us to identify changes in the application that affect end user response times. But let’s have a look at where White Box testing really rocks!
Simple Scalability Analysis
When you want to test how your application scales I recommend using a simple increasing-workload load test. In this test you start with minimum load and ramp it up over time. Using a Black Box approach we analyze the response time of our application. The following illustration shows how our application behaves from a response time perspective:
Looks like our application is doing ok. With increasing load we see response time going up slightly – which can be expected. But – does this mean that this will continue if we add more load on the system? Black Box testing only allows us to assume that the application continues scaling in the same way – but we are not really sure. With White Box Testing we can see into the individual components and can figure out how performance changes in the individual layers.
Getting “White Box” insight into application components while executing a load test allows us to learn more about the scalability of our application. In this case, it seems like our Business Logic layer is by far the worst scaling component in our application when we increase load. Knowing this allows us to a) focus on this component when improving performance and b) allows us to make a decision on whether and how to distribute our application components.
Analyzing Regressions on Component Level
During the development process you also want to verify if code changes have any negative impact on performance. An improved algorithm in one component can improve overall performance but can also have a negative impact on other components. The improvement effort is neutralized by other components that can’t deal with the changed behavior.
Let’s have a look at an example. We have an application that contains Presentation, Business and Database Layers. An analysis of the first implementation shows that the Business Layer has many roundtrips to the database to retrieve objects that hardly ever change. The engineers decide that these types of objects would be perfect for caching. The additional Cache Layer frees resources on the database that can be used for other applications. In the next iteration the business owner decides to change certain aspects of the business logic. This change doesn’t seem to have any negative impact on the response time as reported from the Black Box Tests. When looking at all involved components, however, we see that the changed Business Logic is bypassing the Cache because it requires other objects that have not been configured for caching. This puts the pressure back on the database and is good example for a component level regression.
Where to go from here?
It doesn’t really matter which tools you are using. Whether you use open source or commercial load testing tools or whether you use commercial performance management software like dynaTrace or your own home grown performance logging mechanisms. It is important that you run tests continuously and that you track your performance throughout the development lifecycle. You are more efficient if you have tools that allow you to automate many of these tasks such as automatically executing tests, automatically collecting performance relevant data on code level and also automatically highlighting regressions or scalability problems after tests are executed.
To give you an idea – here are some screenshots of how automated and continuous White Box Testing can help you in your development process.