There is no doubt that performance is important for your business. If you don’t agree you should check out what we and others think about the Performance Impact on Business or remember headlines like these:

The question therefore is not whether performance is important or not. The question is how to ensure and verify your application performance is good enough

Use your End-Users as Test Dummies?

In times of tight project schedules and very frequent releases some companies tend to release new software versions without going through a proper test cycle. Only a few companies can actually afford this because they have their user’s loyalty regardless functional or performance regressions (again – only a few companies have that luxury). If the rest of us were to release projects without proper load testing we would end up as another headline on the news.

Releasing a new version without proper load testing is therefore not the correct answer.

Don’t let them tell you that Load Testing is hard

When asking people why they are not performing any load tests you usually hear things along the following lines

  • We don’t know how to test realistic user load as we don’t know the use cases nor the expected load in production
  • We don’t have the tools, expertise or hardware resources to run large scale load tests
  • It is too much effort to create and especially maintain testing scripts
  • Commercial tools are expensive and sit too long on the shelf between test cycles
  • We don’t get actionable results for our developers

If you are the business owner or member of a performance team you should not accept answers like this. Let me share my opinion in order for you to counter some of these arguments in your quest of achieving better application performance.

Answer to: What is Realistic User Load and Use Cases

Indeed it is not easy to know what realistic user load and use cases are if you are about to launch a new website or service. In this case you need to make sure to do enough research on how your new service will be used once launched. Factor in how much money you spend in promotions and what conversion rate you expect. This will allow you to estimate peak loads.

Learn from your Real Users

It’s going to be easier when you launch an update to an existing site. I am sure you use something like Google Analytics, Omniture, or dynaTrace UEM to monitor your end users. If so, you have a good understanding on current transaction volume. Factor in the new features and how many new users you want to attract. Also factor in any promotions you are about to do. Talk with your Marketing folks – they are going to spend a lot of money and you don’t want your system to go down and all the money wasted.  Also analyze your Web server logs as they can give you even more valuable information regarding request volume. Combining all this data allows you to answer the following questions:

  • What are my main landing pages I need to test? What’s the peak load and what is the current and expected Page Load Time?
  • What are the typical click paths through the application? Do we have common click scenarios that we can model into a user type?
  • Where are my users located on the world map and what browsers do they use? What are the main browser/location combinations we need to test?

The following screenshots give you some examples on how we can extract data from services such as Google Analytics or dynaTrace UEM to better understand how to create realistic tests:

What are the top Landing Pages, the load behavior and page load performance? Testing these pages is essential as it impacts whether a user stays or leaves the web site.
What are the top Landing Pages, the load behavior and page load performance? Testing these pages is essential as it impacts whether a user stays or leaves the web site.
Browser and Bandwidth information allows us to do more realistic tests as these factors impact page load time significantly
Browser and Bandwidth information allows us to do more realistic tests as these factors impact page load time significantly
Analyzing click sequences of real users allows us to model load test scripts that reflect real user behavior
Analyzing click sequences of real users allows us to model load test scripts that reflect real user behavior

CDN, Proxies, Latency: There is more than meets the eye

What we also learn from our Real Users is that not every request makes it to our application environment. Between the End User and the Application we have different components that participate and impact load times: Connection Speed, Browser Characteristics, Latency, Content Delivery Network or Geo Location. A user in the United States on Broad Band will experience a different page load time then a user on a mobile device in Europe that are both accessing an application hosted in the US. To execute tests that take this into consideration you would actually need to execute your load from different locations in the world using different connection speed and different devices. Some Cloud based Testing Services offer this type of testing by executing load from different data centers or even real browsers located around the globe. One example is Gomez Last Mile Testing.

Answer to: We don’t have the tools or the expertise

This is a fair point. As Load Testing is usually not done on a day-to-day basis as it is hard to justify the costs for commercial tools, for hardware resources to simulate the load or for people that need constant training on tools they hardly use.

All these challenges are addressed by a new type of Load Testing: Load Testing done from the Cloud offered as a Service. The benefits of Cloud based Load Testing are

  • Cost Control: you only pay for the actual load tests – not for the time the software sits on the shelf
  • Script generation and maintenance is included in the service and is done by people that do this all the time
  • You do not need any hardware resources to generate the load as it is generated by the Service Provider

Answer to: It’s too much effort to create and maintain scripts

Another very valid claim but typically caused by two facts:

a)      Free vs. Commercial Tools: too often free load testing tools are used that offer easy record/replay but do not offer a good scripting language that makes it easy to customize or maintain scripts. Commercial Tools put a lot of effort into solving exactly this problem. They are more expensive but make it easier, saving time.

b)      Tools vs. Service: Load Testing Services from the Cloud usually include script generation and script maintenance done by professionals. This removes the burden from your R&D organization.

Answer to: Commercial Tools are too expensive

A valid argument if you don’t use your load testing tool enough as then the cost per virtual user hour goes up. An alternative – as you can probably guess by now – are Cloud Based Load Testing Services that only charge for the actual Virtual Users and Time executed. Here we often talk about the cost of a Virtual User Hour. If you know how often you need to run load tests, how much load you need to execute over which period of time it will be very easy to calculate the actual cost.

No actionable data after Load Test

Just running a load test and presenting the standard load testing report to your developers will probably do no good. It’s good to know under which load your application break – but a developer needs more information than: We can’t handle more than 100 Virtual Users. With only this information the developers need to go back to their code, add log output for later diagnostics into the code and ask the testers to run the test again as they need more actionable data. This usually leads to multiple testing cycles, jeopardizes project schedules and also leads to frustrated developers and testers.

Too many test iterations consume valuable resources and impact our project schedules
Too many test iterations consume valuable resources and impact our project schedules

To solve this problem Load Testing should always be combined with an Application Performance Management Solution that provides rich, actionably in-depth data for developers to identify and fix problems without needing to go through extra cycles and in order to stay within your project schedules.

Capturing enough in-depth data eliminates extra test cycles, saves time and money
Capturing enough in-depth data eliminates extra test cycles, saves time and money

The following screenshots show some examples on what data can be captured to make it very easy for developers to go right to fixing the problems:

The first one shows a load testing dashboard including load characteristics, memory consumption, database activity and performance breakdown into application layers/components:

The dashboard tell us right away whether we have hotspots in Memory, Database, Exceptions or in one of our application layers
The dashboard tell us right away whether we have hotspots in Memory, Database, Exceptions or in one of our application layers

In distributed applications it is important to understand which tiers are contributing to response time and where potential performance and functional hotspots are:

Analyzing transaction flow makes it easy to pinpoint problematic hosts or services.
Analyzing transaction flow makes it easy to pinpoint problematic hosts or services.

Methods executed contributed to errors and bad response time. To speed up Response Time Hotspot analysis we can first look at the top contributors …

In-Depth transactional information makes it easy to identify code-level problems
In-Depth transactional information makes it easy to identify code-level problems

… before analyzing individual transactions that have a problem. As every single transaction is captured it is possible to analyze transaction executions including HTTP Parameters, Session Attributes, Method Argument, Exceptions, Log Messages or SQL Statements making it easy to pinpoint problems.

Are we on the same page that Load Testing is important?

By now you should have enough arguments to push load testing in your development organization to ensure that there won’t be any business impact on new releases. I’ve talked about Cloud-based Load Testing services multiple times as it comes with all the benefits I explained. I also know that it is not the answer for every environment as it requires your application to be accessible from the Web. Opening or tunneling ports through firewalls or running load tests on the actual production environment during off-hours are options you have to enable your application for Cloud-based Load Testing.

One Answer to these Questions: Compuware Gomez 360 Web Load Testing and dynaTrace

New Combined Gomez and dynaTrace Web Load Testing Solution provides an answer to all the questions above and even more. Without going into too much detail I want to list some of the benefits:

  • Realistic Load Generation using Gomez First Mile to Last Mile Web Testing
  • In-Depth Root-Cause Analysis with dynaTrace Test Center Edition
  • Load Testing is a Service that reduces in-house resource requirements
  • Keep your costs under control with per Virtual User Hour billing
  • Works throughout the application lifecycle – from production, to test, to development

Running a Gomez Load Test allows you to execute load from both Backbone Testing Nodes as well as Real User Browsers located around the world. Especially the Last Mile is an interesting option as this is the closest you can get to your real end users. The following screenshot shows the Response Time Overview during a load test from the different regions in the world allowing you to see how performance of your application is perceived in the locations of your real end users:

World Map gives a great overview how pages perform from the different test nodes
World Map gives a great overview how pages perform from the different test nodes

From here it is an easy drill down in dynaTrace to analyze how increasing load affects performance as well as functional health of the tested application:

There is a seamless drill option from the Gomez Load Testing Results to the dynaTrace Dashboards
There is a seamless drill option from the Gomez Load Testing Results to the dynaTrace Dashboards

Find out more about Gomez 360 Load Testing for Web, Mobile and Cloud Applications.