As part of our series with performance testing expert and author of Web Load Testing for Dummies, Scott Barber provides answers to your load testing questions.
Scott is the founder and Chief Technologist of PerfTestPlus, Inc. and is viewed by many as the world’s most prominent thought-leader in the area of software systems performance testing. He is also a respected leader in the advancement of the understanding and practice of testing software systems. Read more about Scott at his website.
In the Part one, two, and three of this series, Scott provided details on how to establish performance goals and targets, and when and how to performance test. In the final post of this series, Scott answers your testing execution and tools questions.
Q. We run each test case 3 times, first number will be un-cached, it takes more time and the other 2 runs will be cached so takes bit less time. We then take the average of 3 runs, it this a good process?
SB: Don’t average them, report them separately. That is, if you’re trying to assess the difference in response time and/or system stress related to first time vs. returning users. After you’re satisfied with that assessment, find the setting in your tool that allows you to set the % of users to simulate as first time vs. returning and set the % to a number appropriate to the production situation you are trying to simulate.
Q. Can you get the same testing results when different tools are used?
SB: If you can’t, that’s a problem. If you are measuring the same thing (not always easy to do – sometimes it takes a *lot* of custom configuration, sometimes it requires all but reverse engineering the tool to figure out what it’s measuring in the first place) and your simulations are identical (from a *very* detailed server-side perspective), etc, etc, etc, and you are getting different results from different tools, that is bad (unless, of course, one or more of the tools you are trying to compare doesn’t actually support what you are trying to do).
In all of the side-by-side tool comparisons I’ve done over the years, I have only ever worked with one whose results I could simply *not* replicate with *any* other tool – and that tool has been off the market for years.
There are tools out today that I have *not* done that kind of analysis on, so if there is one (or more) out there that’s spitting out results that are different than the other tools, I’d like to know which tool, and I’d like to review the test(s) you used to come to that conclusion, ‘cause if your analysis is correct, I’d like to have a little talk with the creator/vendor for that tool.
Q. Could you please recommend some books about load testing?
SB: In addition to Web Load Testing for Dummies, I’d suggest starting with the ones that interest you from http://www.perftestplus.com/books.htm , but except for Performance Testing Guidance for Web Applications, An Introduction to General Systems Thinking, and both of Steve Souder’s books, High Performance Web Sites: Essential Knowledge for Front-End Engineers and Even Faster Web Sites: Performance Best Practices for Web Developers, I’d recommend sticking with blogs & articles.
The books are great, but they’ll only take you so far, and when it comes to Performance Testing, that’s not very far – and I say that as the leading author of the most popular book dedicated to performance testing software ever written (which is roughly like claiming that I’m the weightlifting champion of my household – it’s true, just not overly impressive as my sons are 12 & 8 and Dawn doesn’t lift weights).
My point is, what you’re going to find inside the flaps of a book with an ISBN is either going to be extremely specific and narrow, or pretty general.