Website Performance: Designing a Test Strategy to Evaluate CDN Performance

Moving static content to a CDN (Content Delivery Network) is a proven and relatively simple way to improve perceived website performance.  By using a CDN to cache data on servers near the “edge” of the internet, you will not only enable faster access for the user, but also reduce your data center requirements.

The easiest and least expensive content to move to the edge is static data – images and text files that change infrequently.   But how do you decide which CDN to use?

The ability of CDNs to deliver content fast is dependent on their dynamic routing and caching algorithms as well as the locations of each CDN’s servers in relation to the locations of site visitors.  A well-designed test strategy can help to determine how much improvement is to be expected and which CDN does the best job of delivering it.

We’ve identified some best practices to consider when designing a testing strategy. These are derived from customer CDN comparison assessments:

1. Measure What Matters

It may be tempting to use a few single, static images to compare CDN performance in efforts to eliminate the effects of all other variables except the CDN itself.  For an ad server, whose goal is to deliver single objects as fast as possible and then move on, this may be a valid test.

However, for any business that needs to deliver online transactions (bank login or retail checkout for example), this simple test won’t tell you how a CDN affects the end user’s experience.  Instead, you want the test to replicate an important business transaction traversing multiple pages.  A multi-step transaction will help to confirm that caching has been implemented properly.

It’s important to understand not only how much the CDN affects response time for single objects, but also how it affects the overall user experience.  Why accelerate the static content if the overall experience is essentially unchanged?

If your site’s performance is heavily influenced by dynamic content, then accelerating static content with a CDN may not provide the expected boost in website performance.  Third party content continues to be delivered from the third party, so a CDN won’t help you there, either.

2. Measure from Where Your Users Are

We’ve found that the most effective CDN comparison tests employ a combination of Backbone and Last Mile measurements.  Backbone nodes, located in ISP data centers with high-bandwidth internet access and top-notch servers provide a lab-like environment for consistent measurements.   The Backbone data can provide insight into the relative effectiveness of each CDN’s web acceleration strategy and technology.

However, the cleverness of the CDN’s solution is not what you are paying for – you’re paying for better website performance for the end user.   Last Mile data, collected from peers installed on real users systems on a wide variety of desktops and laptops, located on different ISPs with different bandwidths, provides a view of the real user experience at the end of the application delivery chain.  In a recent CDN comparison, one of the CDNs “failed” because they did not configure their environment correctly.  They actually delivered content to the end user more slowly than the customer’s server.  In the Backbone test, their content delivery was on par with the customer’s server, but by measuring from the Last Mile, the slowdown was easily identified.

Both perspectives – the lab-controlled environment of the backbone and the real-world experience of the Last Mile – are needed for a complete picture of CDN performance.

3. Minimize Extraneous Variables

We cautioned against using an artificially simple case to control variables, however it’s also important to control certain variables in order to get a valid comparison.

As a first step, line up any prospective CDNs for a simultaneous bake-off.  Conditions on the Internet can change over time – an outage at an ISP’s data center one week will be fixed by the next.   It may be tempting, due to scheduling challenges, to test one CDN starting today, and the next candidate starting a week or two down the road.

But if you engage the CDN prospects serially, they are guaranteed not to be tested under equivalent conditions, due to random variations in the internet.   Instead, allow enough time during the planning phase to get all CDNs on board and configured to run performance tests during the same time period.  (A minimum of one week is required, and two weeks are recommended, to capture the effects of any time-of-day or day-of-week variations.)

Not only is it important to test all the prospective CDNs during the same time period, but you also need to take steps to make sure the individual tests  run from the same locations at (almost) the same time.  This is straightforward in Backbone testing where you simply deploy your CDN tests to run on the same Backbone nodes with the same frequency.

You can achieve that same level of control in Last Mile testing by “batching” your CDN tests (one test per CDN) so they are treated as a unit. If the tests are not batched, they may be assigned to run on different peers that would not be matched for location, ISP, or other variables found on end-user systems.  If the tests are batched, then they are assigned as a group to run sequentially on the same peer, thus controlling for those end-user variables.

Your users’ proximity to your web servers has a significant impact on response times. Content delivery networks (CDNs) let you deploy your content (or entire web site) across multiple, geographically dispersed servers to make your pages load faster for end users. However, companies need to choose CDNs carefully.  One critical aspect of the selection process is to objectively measure CDNs’ performance. By using the best practices above when designing a testing strategy, you can best determine if a CDN can provide a significant improvement in website performance for all your users.

What are your experiences with selecting a CDN to improve website performance?