There is no doubt that performance is important for your business. If you don’t agree you should check out what we and others think about the Performance Impact on Business or remember headlines like these:
- Target.com web site was down after promoting a new labels: Article on MSN
- Twitter was down and people complaining about it on Facebook: Huffington Post Article
- People stranded on Airports because United had a software issue: NY Time Article
The question therefore is not whether performance is important or not. The question is how to ensure and verify your application performance is good enough
Use your End-Users as Test Dummies?
In times of tight project schedules and very frequent releases some companies tend to release new software versions without going through a proper test cycle. Only a few companies can actually afford this because they have their user’s loyalty regardless functional or performance regressions (again – only a few companies have that luxury). If the rest of us were to release projects without proper load testing we would end up as another headline on the news.
Releasing a new version without proper load testing is therefore not the correct answer.
Don’t let them tell you that Load Testing is hard
When asking people why they are not performing any load tests you usually hear things along the following lines
- We don’t know how to test realistic user load as we don’t know the use cases nor the expected load in production
- We don’t have the tools, expertise or hardware resources to run large scale load tests
- It is too much effort to create and especially maintain testing scripts
- Commercial tools are expensive and sit too long on the shelf between test cycles
- We don’t get actionable results for our developers
If you are the business owner or member of a performance team you should not accept answers like this. Let me share my opinion in order for you to counter some of these arguments in your quest of achieving better application performance.
Answer to: What is Realistic User Load and Use Cases
Indeed it is not easy to know what realistic user load and use cases are if you are about to launch a new website or service. In this case you need to make sure to do enough research on how your new service will be used once launched. Factor in how much money you spend in promotions and what conversion rate you expect. This will allow you to estimate peak loads.
Learn from your Real Users
It’s going to be easier when you launch an update to an existing site. I am sure you use something like Google Analytics, Omniture, or Dynatrace UEM to monitor your end users. If so, you have a good understanding on current transaction volume. Factor in the new features and how many new users you want to attract. Also factor in any promotions you are about to do. Talk with your Marketing folks – they are going to spend a lot of money and you don’t want your system to go down and all the money wasted. Also analyze your Web server logs as they can give you even more valuable information regarding request volume. Combining all this data allows you to answer the following questions:
- What are my main landing pages I need to test? What’s the peak load and what is the current and expected Page Load Time?
- What are the typical click paths through the application? Do we have common click scenarios that we can model into a user type?
- Where are my users located on the world map and what browsers do they use? What are the main browser/location combinations we need to test?
The following screenshots give you some examples on how we can extract data from services such as Google Analytics or Dynatrace UEM to better understand how to create realistic tests:
CDN, Proxies, Latency: There is more than meets the eye
What we also learn from our Real Users is that not every request makes it to our application environment. Between the End User and the Application we have different components that participate and impact load times: Connection Speed, Browser Characteristics, Latency, Content Delivery Network or Geo Location. A user in the United States on Broad Band will experience a different page load time then a user on a mobile device in Europe that are both accessing an application hosted in the US. To execute tests that take this into consideration you would actually need to execute your load from different locations in the world using different connection speed and different devices. Some Cloud based Testing Services offer this type of testing by executing load from different data centers or even real browsers located around the globe. One example is Gomez Last Mile Testing.
Answer to: We don’t have the tools or the expertise
This is a fair point. As Load Testing is usually not done on a day-to-day basis as it is hard to justify the costs for commercial tools, for hardware resources to simulate the load or for people that need constant training on tools they hardly use.
All these challenges are addressed by a new type of Load Testing: Load Testing done from the Cloud offered as a Service. The benefits of Cloud based Load Testing are
- Cost Control: you only pay for the actual load tests – not for the time the software sits on the shelf
- Script generation and maintenance is included in the service and is done by people that do this all the time
- You do not need any hardware resources to generate the load as it is generated by the Service Provider
Answer to: It’s too much effort to create and maintain scripts
Another very valid claim but typically caused by two facts:
a) Free vs. Commercial Tools: too often free load testing tools are used that offer easy record/replay but do not offer a good scripting language that makes it easy to customize or maintain scripts. Commercial Tools put a lot of effort into solving exactly this problem. They are more expensive but make it easier, saving time.
b) Tools vs. Service: Load Testing Services from the Cloud usually include script generation and script maintenance done by professionals. This removes the burden from your R&D organization.
Answer to: Commercial Tools are too expensive
A valid argument if you don’t use your load testing tool enough as then the cost per virtual user hour goes up. An alternative – as you can probably guess by now – are Cloud Based Load Testing Services that only charge for the actual Virtual Users and Time executed. Here we often talk about the cost of a Virtual User Hour. If you know how often you need to run load tests, how much load you need to execute over which period of time it will be very easy to calculate the actual cost.
No actionable data after Load Test
Just running a load test and presenting the standard load testing report to your developers will probably do no good. It’s good to know under which load your application break – but a developer needs more information than: We can’t handle more than 100 Virtual Users. With only this information the developers need to go back to their code, add log output for later diagnostics into the code and ask the testers to run the test again as they need more actionable data. This usually leads to multiple testing cycles, jeopardizes project schedules and also leads to frustrated developers and testers.
To solve this problem Load Testing should always be combined with an Application Performance Management Solution that provides rich, actionably in-depth data for developers to identify and fix problems without needing to go through extra cycles and in order to stay within your project schedules.
The following screenshots show some examples on what data can be captured to make it very easy for developers to go right to fixing the problems:
The first one shows a load testing dashboard including load characteristics, memory consumption, database activity and performance breakdown into application layers/components:
In distributed applications it is important to understand which tiers are contributing to response time and where potential performance and functional hotspots are:
Methods executed contributed to errors and bad response time. To speed up Response Time Hotspot analysis we can first look at the top contributors …
… before analyzing individual transactions that have a problem. As every single transaction is captured it is possible to analyze transaction executions including HTTP Parameters, Session Attributes, Method Argument, Exceptions, Log Messages or SQL Statements making it easy to pinpoint problems.
Are we on the same page that Load Testing is important?
By now you should have enough arguments to push load testing in your development organization to ensure that there won’t be any business impact on new releases. I’ve talked about Cloud-based Load Testing services multiple times as it comes with all the benefits I explained. I also know that it is not the answer for every environment as it requires your application to be accessible from the Web. Opening or tunneling ports through firewalls or running load tests on the actual production environment during off-hours are options you have to enable your application for Cloud-based Load Testing.
One Answer to these Questions: Compuware Gomez 360 Web Load Testing and Dynatrace
New Combined Gomez and Dynatrace Web Load Testing Solution provides an answer to all the questions above and even more. Without going into too much detail I want to list some of the benefits:
- Realistic Load Generation using Gomez First Mile to Last Mile Web Testing
- In-Depth Root-Cause Analysis with Dynatrace Test Center Edition
- Load Testing is a Service that reduces in-house resource requirements
- Keep your costs under control with per Virtual User Hour billing
- Works throughout the application lifecycle – from production, to test, to development
Running a Gomez Load Test allows you to execute load from both Backbone Testing Nodes as well as Real User Browsers located around the world. Especially the Last Mile is an interesting option as this is the closest you can get to your real end users. The following screenshot shows the Response Time Overview during a load test from the different regions in the world allowing you to see how performance of your application is perceived in the locations of your real end users:
From here it is an easy drill down in Dynatrace to analyze how increasing load affects performance as well as functional health of the tested application:
Find out more about Gomez 360 Load Testing for Web, Mobile and Cloud Applications.