DevOps Survival Guide: 2013 Online Holiday Shopping Season

The online holiday shopping season is approaching faster than most teams that are developing, testing and running eCommerce sites like. In this blog we summarize the feedback from our eCommerce customers into what they consider a DevOps Survival Guide for the 2013 Online Holiday Shopping Season. The ultimate goal is high conversion rates, followed by implementing changes based on lessons learned from 2012. Additionally, this guide will help teams cope with the predicted explosion of mobile online shoppers.

Top Goal for 2013: High Conversion Rates

It is the shared responsibility of every developer, tester, operations, business and marketing team to work towards achieving high conversion rates. Developers, Testers and Operations can obviously not impact the products that are sold or how they are marketed in campaigns – but – they can make sure that those visitors who are driven to the website have a good user experience and stay on the site to actually convert to a customer. In the end, it is the customer that drives revenue and business growth and ultimately makes sure all of them get paid at the end of the month.

Monitoring other Key Metrics such as Real vs. Bot Traffic, Geographical and User Type Information
Monitoring other Key Metrics such as Real vs. Bot Traffic, Geographical and User Type Information

What Impacts Conversion Rates?

Let’s start by looking at what actually impacts the conversion rate. There are several reasons why visitors drop out that are “under the control” of the DevOps Teams:

  • Slow Response Times: End Users are spoiled with performance. They abandon the site if they have to wait too long for downloading product details or waiting for the checkout to be completed.
    Actions: Trim and optimize your page load; provide enough bandwidth; leverage CDNs for faster content deliver and actively monitor you applications, the infrastructure and 3rd party services. Read up on Supersized Content and Control Page Load Time.
  • Failures: Broken functionality due to JavaScript errors or bugs in the server implementation lead to frustration. Most visitors will not try a different browser or device to re-run through all steps – they will just switch to a competitor
    Actions: Don’t add every JavaScript gimmick to the site; choose frameworks that work reliably across all platforms; test on different platforms and browsers, and provide a “back door” to turn off certain features for specific browsers. Learn more on JavaScript Bugs in Safari or 5 Things Learned from Strong Performers in 2012.
  • Complete or partial Outages: When following a promotional link from an email or social network leads to “unexpected” high traffic volume causing the site to crash or respond very slowly you will see visitors abandon that link and not try a second time. That is waste of marketing money
    Actions: Sync with marketing activities and plan additional resource allocations ahead of time in order to avoid what is called a “Marketing Denial of Service” MDoS. Also sync with your 3rd party vendors as they also need to handle the load. Also look into Lessons learned from Black Friday 2012 Outages and The Ripple Effect of Facebook Outage.

Improve Conversion by learning from 2012

Good advice is to look back and fix things that impacted conversion rates in 2012. Here is a list of issues we heard from our customers, as well as issues we blogged about during and after the previous holiday season:

  • 3rd Party Impact on Performance: Ad-Services, Social Network Plugins, Payment Services, CDNs, etc. – all of these 3rd party services that are used by a large number of eCommerce sites see more traffic than any individual eCommerce site. To no surprise it is more likely that the 3rd party’s bad performance impacts all companies that use their services.
    Actions: Include 3rd party and CDNs in your testing; implement your site so that 3rd party content doesn’t impact load time of your own content and make sure to monitor 3rd Party Performance. Implement a “bail out” option to turn off 3rd party content to avoid a major performance impact. More Tips? Controlling your 3rd Party Content and Getting Visibility into Akamai,
  • Mobile taking over the Business: Based on predictions, more than 50% of shoppers will shop on their mobile devices this year. This requires special focus in development to optimize content for tablets and mobile devices as well as extending testing to those commonly used devices.
    Actions: For mobile native apps it is important to have a fallback strategy, e.g.: switching to web views in case your mobile native app has problems and you can’t wait for days or weeks until the fixes are approved by Apple. Also – make sure you do proper testing across the most popular devices and browser versions. Read more on Don’t miss 50% of your paying end users
  • Lack of “basic” WPO: Especially on landing and product pages it is important to focus on basic WPO such as content compression, caching, or using CDNs. In the chaos of events these simple and basic best practices too often are forgotten.
    Actions: Check at least your landing and product pages with tools such as Dynatrace AJAX Edition, YSlow, or Page Speed and optimize based on their recommendations. Check out our blog posts on Top Performance Landmines
  • Predicting Traffic: What were the traffic predictions vs. actual load? What can you learn for the predictions this year? How much load do you need to sustain and test for?
    Actions: Sync with business and base your new predictions on historic data.

Measuring Conversion Rates

Conversion Rates are the number of visitors on a site and the percentage of those that actually buy something. Whereas the “Overall Conversation Rate” and the generated revenue is the ultimate measure we want to look for.  It is as important to look at the individual steps along the so called Conversion Funnel, e.g.: Product Page, Add to Cart, Proceed to Checkout, Enter Billing Information, etc.

Measuring the funnel steps allows you to take action in case the Conversion Rate is low or dropping as it tells you that e.g.: most people drop out when entering payment information and the most likely reason is a 50% failure rate when calling the credit card authorization service.

For every step in the funnel, it is recommended that one measures # of visits, response time, failure rate, # of drop outs similar to what is shown on the following screen (for more information also read the following blog):

eCommerce dashboard correlating performance with number of visits along the funnel.
eCommerce dashboard correlating performance with number of visits along the funnel.

Is there a difference between iOS, Android, Wi-Fi, 3G connections or the screen resolution?

To take it one step further it is recommended to you look at these conversion funnel metrics per end user category where category includes the type of the device (desktop, table, mobile phone), the connectivity (Wi-Fi, 3GB, Broadband), the screen resolution and the geographical region. This allows you focus your triage to e.g.: fix problems that happen on devices with smaller screen resolutions or a slower connection.

Measuring Landing Pages, Exit Pages & Co

There are a lot of other metrics that can tell you how well your site is doing:

  • Landing Pages: The landing page is the first impression that visitors get. It is important to optimize and monitor the performance and failure rates of all major landing pages. That includes the home page but also special promotional landing pages. This allows you to immediately react to increased bounce rates by e.g.: adding additional hardware or fixing problems.
  • Exit Pages (Dropout Pages): Besides the conversion funnel pages you also want to monitor all other pages where visitors leave the site. There may be pages that didn’t seem important in the first place but might be a problem for your real end users.
  • Type of Visitors: It is important to analyze all performance aspects of the used device, OS and browser. If you see a problem that only happens on iOS it is easier to focus the right people on the root cause analysis.
  • Excluding Bots or Bogus Load: More often we also see Bots and other synthetic generated load that needs to be excluded from the general performance measures. It is important to monitor the activity and you should also have the ability to “block.”
  • Geographical Region: Knowing the geographical region of your visitors allows you set priorities when it comes to error analysis. Visitors from your main target markets should have a higher priority than visitors that just stumble across your website that are not going to buy something anyway

Bounce Rates: It is important to analyze bounce rates by devices. A faulty CSS or JS file that causes a problem on a specific browser version may result in a higher bounce rate than average. Bots and bogus load also heavily impact your bounce rates as they often only load a single page which could impact your bounce rate. Make sure to exclude these in your calculation.

Monitoring other Key Metrics such as Real vs. Bot Traffic, Geographical and User Type Information
Monitoring other Key Metrics such as Real vs. Bot Traffic, Geographical and User Type Information

Implementation Guidelines for 2013

Knowing what went wrong in 2012 and which metrics to look at to ensure higher conversion rates is a start. The next steps are to do proper testing before you enter the holiday season and to get your production system ready to tell you all the metrics you need throughout the holiday season.

Therefore stay tuned for two follow up blog posts where we talk about specific implementation guides for developers, testers and operation teams where we cover the following items:

Step 1: With Confidence into the Holiday Season: Verifying Readiness in Test / Pre-Prod

  • Ensure quality of key pages along the conversion funnel
  • Can we handle the load? Can our CDNs / 3rd Party components handle the load?
  • Setup and verify monitoring and alerting for production
  • Testing from real end user locations + all devices!

Step 2: With Confidence through the Holiday Season: Manage Success in Production

  • How to report business success to management: Conversion Rates, Funnel, etc.
  • What can go wrong and how to proactively fix problems?
  • Monitor your SLAs with a combination of real and synthetic user monitoring
  • How to monitor and react to 3rd party outages

How to monitor your mobile user and what to do when your mobile app breaks

Klaus is a Senior Technology Strategist in the Center of Excellence at Dynatrace. Klaus influences the strategic direction and development of application performance management solutions. He has deep experience gleaned from years of developing and running large scale web and mobile applications for online businesses.