The following is Part one of a series that will take place over the next several weeks with performance testing expert and author of Web Load Testing for Dummies, Scott Barber, answering your web load testing questions.
Scott is the founder and Chief Technologist of PerfTestPlus, Inc. and is viewed by many as the world’s most prominent thought-leader in the area of software systems performance testing. He is also a respected leader in the advancement of the understanding and practice of testing software systems.
Let’s get started on the questions:
Q. On gathering the project team to determine the goals of the performance testing effort: What if the members of the team you gather know nothing about performance testing? Are there any tips to help get needed information from them?
SB: First, you want them to tell you what they want to learn as a result of doing performance testing qualitatively, not quantitatively. For example:
– “Will this version of the site be faster than the last version?”
– “Will it be faster than our hated competitor’s site?”
– “Will we be the ones in the news on black Friday because our site crumbled under the load?”
– “Will our users complain when they try to use our app on a 3G network?”
– “Can we set up alerts in production to give us early warning before it starts too slow?”
– “Can you tell me what component we’ll need to upgrade first if we start getting a lot more traffic?”
– “Can we help the developers optimize their code early so we don’t have a problem like last time?”
– “Will our current production hardware support this app at the projected volume? If not, how much more do I need to buy?”
If you can collect questions like this, you can convert them into goals by turning the question into a statement, like:
- Speed goal: Faster than hated competitor X
- Volume goal: Withstand black Friday load
As for putting numbers to these goals, it’s just a matter of doing your homework. For almost all of these, a few minutes with your favorite search engine, or a coffee & doughnut for your favorite IT/Ops member and you’ll have your numbers.
The only number that is actually difficult to come up with is “target load”. For that, you’re going to have to listen to what your stakeholders are predicting (in terms of sales, or total site members, or searches, or posts – whatever metric the sales team uses in their reports to the CEO) and work backwards to come up with a number that makes sense. Don’t be afraid to get some help, and don’t be afraid to use the resources I mention below to help you out.
Q. We need to search for baseline using the current application measures, right? If we have a new application, it obviously will have neither historical data nor sales projections and so on. In that case, how do we decide on the target load? Apart from looking at historical data of similar applications, is there any other way?
SB: Not necessarily. If there is a previous version of the application, collecting performance related data from both the production environment, and from the final set of performance tests you conducted on that version can be very useful when you have a goal of “performance that is no worse than the previous version”, there are plenty of other methods you can use to quantify your performance targets.
The easiest thing to do is to have key stakeholders refer you to a site or an application that exhibits performance characteristics they like. From there, you can easily paste the URL into a free online analyzer (like http://www.websiteoptimization.com/services/analyze/) or install a browser plug in (like YSlow http://developer.yahoo.com/yslow/ or PageSpeed http://developer.yahoo.com/yslow/) and simply navigate to the site and view the analysis.
Of course, these numbers won’t be perfect, but neither will your historical data if you are making any user noticeable changes to the site or application.
The reality is that you don’t actually need to be all that close on your target load. What you need is a number to work toward. The system isn’t going to handle that number (whatever it is) when you start running your tests – it’s probably not going to handle that number when the stakeholders decide to ship it – and besides, odds are that you’re not going to be testing on the production system anyway, so whatever number you get up to in your test environment is going to be different than what you’re going to see in production anyway.
So make your best guess, and the less confident you are about the accuracy of that number, the bigger number you multiply that best guess by. As long as you have a number that is between accurate and bigger than accurate that the team agrees is *the* target number (until the *team* agrees to change the target number to something else), then you can stop worrying about the number, start testing, and start worrying about all the performance issues keeping you from achieving that number.
Notice that during the webinar, nor in the Web Load Testing for Dummies book, nor in this post, nor in anything I have written since 2003 do I use the phrase “performance requirement” except when there is a number defined in a legally enforceable contract, or a number equates to “achieve this number or real human beings will be injured or die” (for instance, the response time of an anti-lock braking skid sensor).
Everything else is a goal, a target, or what I like to call “desirements”, because when it comes to performance, we all know that the stakeholders are going to decide to ship when they decide they are losing more money by not shipping than they think they will lose by shipping with whatever the performance numbers are today. After that, it’s a monitoring and tuning issue. If you’ve done your job well, you’ll be able to tell the folks doing the monitoring to keep an eye out for so they can see the big slowdowns coming in advance and be proactive in minimizing the impact.
(I’m not specifically promoting these sites over others like them, these are simply the first ones that came to mind. These free tools are designed for websites. If you’re not testing a website, use your favorite search engine to find free analyzers for the type of application you’re interested in. Lots of companies make the basic functions of their analyzers available for free, or have free trial periods)
Feel free to submit a question on load testing and stayed tuned for Part two!