Scott is the founder and Chief Technologist of PerfTestPlus, Inc. and is viewed by many as the world’s most prominent thought-leader in the area of software systems performance testing. He is also a respected leader in the advancement of the understanding and practice of testing software systems. To read more about Scott, visit http://about.me/scott.barber
In Part one of this series,Scott provided details on how to establish performance goals and targets. This next set of questions addresses these topics in more detail:
Q. How does setting goals improve performance criteria?
SB: Using these methods [from Part one] to establish performance criteria can’t help but improve the criteria. Only twice in my 10 years, hundreds of clients, and *thousands* of stories from performance testers do I know of a stakeholder providing quantified performance criteria that was both valuable and accurate.
Once it came from a VP who (I later found out) had spent 16 years specializing in performance optimization of mainframes for client server applications before he moved into management. The other time it came from the capacity planning team (which is where at least some of those numbers *should* be coming from, but this was the one time that I was working with a client where the capacity planning group was both far more mature than the performance testing group *and* happy to collaborate).
The only other times I can imagine getting hard numbers from stakeholders that are correct and valuable are when they come from Service Level Agreements or when the application is to be deployed on some very specific and pre-existing hardware platform (so, they might say, “This application cannot consume more than 4G of RAM on the MQ box, ‘cause that’s all we’ve got & I’m not buying another one this year!”)
Q. How do we specify the target goals and probability percentage of each page to be hit?
SB: For modeling the load (specifying the percentages) the most important thing to remember is that unless you have lots of dependable, actual usage statistics, these percentages are somewhere between a swag (scientific wild a** guesses) and a shot in the dark – which is why it is absolutely critical that you conduct tests at the same load level with at least 3 different distributions of those percentages.
I always try to be as accurate as I can without losing a single minute of actual testing time with the first distribution. I call that one “expected”. Then I create 2 more distributions that I call “easy performance day” and “hard performance day”. For the easy day, I make the percentages extra large on all the activities that I think have minimal performance impacts on the system and make the percentages extra small on the ones that I think have major performance impacts on the system. For hard day, I do the opposite.
I depict these distributions using the User Community Modeling Language (UCML™) to make it easier for others to understand specifically what I mean when I’m explaining the differences among these three models.
When I execute a test at using these 3 different distribution models, what I hope to see is no difference in any performance measure I care about because that means that if my model is off by up to that much, I’m still safe. What I usually see is that the site or application can handle much more load with the easy day scenario without slowing down than it can with the hard day scenario. That’s ok. In fact, that is *exactly* what I report. For example:
“We can handle 5k with the expected workload – give or take about 750 depending on just how far off our estimates of the workload distribution were”
*Note* I’m not trying to be selfish by pointing to only my work to answer this question. It’s just that the question relates to using UCML™ that Chris Walters and I invented on a white board about 10 years ago. There are other methods available, but if you have no historical data, it’s all a matter of estimation anyway.
Also refer to Chapter 12: Modeling Application Usage of Performance Testing Guidance for Web Applications by MS patterns & practices (don’t worry, you can browse it on the web, download it as a .pdf, or buy it if you like paper. I am very proud that one of my stipulations to being a co-author of the book was that the content would be made forever free and public on the web) and the associated video here http://bit.ly/o0pDUT.
Here’s a link to my Google TechTalk http://bit.ly/qtwpnx and you can also review dozen articles and presentations on my site (specifically on the pages http://scott-barber.blogspot.com/p/articles-columns-papers-books-by-scott.html and http://scott-barber.blogspot.com/p/slides-multi-media.html if you are looking for anything related to UCML, models, or modeling.
Q. What skills are necessary for a Performance Engineer (NOT a Performance Tester) and what are the expectations (by Management & Project Technical Teams) for a Performance Engineer?
SB: In many ways, these are entirely impossible questions to answer. Why? Because if I were to, for example, ask on Twitter “What’s the difference between a performance tester and a performance engineer?” I am 100% confident that there would be a set of answers ranging from “one gets paid more”, to “they are the same roles in different companies”, to “testers test, engineers tune”, to “who cares?”
Having said that, I’m going to make guess as to the distinction you are getting at, which is as follows:
Performance Testers write and execute scripts. They might do some test design and some analysis. They don’t lead teams. They don’t tune or make specific tuning recommendations. Basically, they are tool specialists who know how to generate load and collect data, but aren’t trusted to make technical or business decisions.
Performance Engineers lead teams. They interface with management, development, IT/Ops and others. They are key members of architecture, analysis, and tuning teams. These are “Architect-level” application performance specialists, who “can do it all”, including Software Performance Engineering (SPE www.perfeng.com/) and Capacity Planning (at least to some degree).
Of course, simply defining terms pretty well answers your questions as best I know how without actually asking “Higher Management & Project Technical Team” in your organization that question.
Personally, I think of a “Performance Engineer” as a person who is implementing SPE… of course, most of the people I have met who are implementing SPE go by the title “Architect”, thus making the whole distinction kind of pointless.
Part three in this series will address when and how to performance test.