Advance Planning for Your Holiday Load Testing

It’s September which means its crunch time for holiday website traffic planning.  I thought I’d take a moment or two to suggest some areas to focus on if you’re still preparing your website for the upcoming holiday season. This is the first post in a series of three that looks to provide some insight and web load testing best practices in preparation for the busy online holiday season. This first post looks at what you should be focusing on if you’re still in the planning stages of holiday load testing.

The thing to focus on is planning.  Even this close to the holidays – you shouldn’t just wake up one morning and decide to web load test your site.  You want to plan it out so that you test the right things and if your testing cycles are short or not as extensive as you hoped, you test what is most important.

The second area of focus helps you with the planning.

Get the whole team on board.  By the whole team I don’t mean all of your QA test engineers.  I mean everyone with a stake in the holiday website, the list should include a representative from Marketing, It operations, Development, possibly a business manager or PR person and of course the QA team.

Both business and IT must provide input into the testing goals and success criteria as well as the test plan and methodology, the business is in the best position to estimate the expected traffic and sales growth or campaign information and the IT team needs to unsure that the business understands the capabilities and limitations of the environment and the level of effort and cost to meet the growth demands.

Getting  the whole team in a room to hash out what has changed on the website, what the expectations are for revenue generation, traffic spikes, and customer performance is how you move on to the third area of focus.

Establish goals. Each team member from each department will have goals for the web application.  Additionally, you have to think about your customer’s goals.  You need to test from the customer’s perspective, and test what they want to do.  That is how you establish your testing goals.

Some general questions to consider in establishing goals are:

  • Will the site break under anticipated peak load? Can we handle unanticipated surges in traffic?
  • Will our most important transactions and pages perform properly- regardless of load?
  • Will our customers have a positive experience no matter where they are located geographically or what device they may be using?
  • Are our site’s third-party providers such as rating and reviews, ads, nes feeds, ecommerce engines or CDNs hurting our performance?

Figure out what to measure. Based on the team’s input you can narrow your focus on what is important to test – what metrics you need to obtain. Some common measurements include:

  • User perceived response time
  • Response time vitiations by geography
  • Number of virtual users- measured at many points in time
  • Resource utilization such as CPU Memory, Disk I/O, network bandwidth
  • Number of errors

Approaching your testing in this planned manner ensures that the right input is taken in so that the output means something.  In our experience helping many customers prepare for  the holiday season, we have witnessed the IT team moving forward with load testing without direct involvement from the business, whether due to corporate culture, lack of knowledge or perceived lack of time.

The tests can succeed – in a vacuum- but until the site is launched and the volume of users peaks the company has no way of knowing if they really have a site that will support their users.  This isolated approach causes a good deal of unnecessary stress and any time saved by not have a couple meetings is easily wiped out if the testing doesn’t test the right things.

In my next post, I’ll walk through an actual customer success story for a large online retailer. For more insight into preparing for holiday web load testing, watch this short video “Lessons Learned during the Holiday Shopping Season.”