Whether you are a Fan of US Football or not – it was really hard to avoid this huge sport event on February 5th. Besides the actual game it is the Super Bowl commercials that – besides being very expensive to air – usually drive a lot of load on the websites of the companies that run their ads. The question is whether the millions of dollars spent really drive consumers to these websites and make them do business with them.
As we won’t get an answer from the top brands that advertised about the actual conversion rates we can look at End User Experience and Web Site Performance of their web sites while these Ads were aired. By analyzing this data that we can get through continuous synthetic monitoring combined with deep dive browser diagnostics we will be able to see whether their web application was actually able to handle the load and didn’t leave too many of these users with a bad user experience.
Use Case: What was the User Experience of One of the Top Brands during Super Bowl Day
In order to avoid any discussions on whether the numbers we present are good or bad and mean that this company did a good or terrible job we present all this data without giving an actual name as the purpose of this exercise is to show how to perform this type of analysis.
Synthetic Monitoring of a Web Site
In order to monitor web site performance of a site we setup a Gomez Synthetic Monitor to be executed on a scheduled basis. The monitor not only monitored the performance of the initial home page but also walked through several main use case scenarios on that page, e.g: searched for a product. These tests were also executed from multiple different US locations, using Low and High-Bandwidth connections and also using a mixture between Internet Explorer and Firefox browser. Additionally we captured individual performance sessions using dynaTrace Browser Diagnostics.
The following screenshot shows the response time of the Home Page of this web site via our Last Mile platform via a Firefox browser in the days before the Super Bowl as well as during the Super Bowl. The average Page Load time was around 9 seconds with the exception of the timeframe during the Super Bowl – that’s when it jumped up to 30 seconds:
Analyzing Page Load Time from Monitoring Data
There were two factors that drove this spike:
- Higher Load on their application due to the commercial aired during Super Bowl
- Additional Content on their page that doubled the page size
Factor 1: Higher Load
That’s of course intended and a success for their marketing campaign. What you want to make sure is that you can handle the additional expected load by using CDNs (Content Delivery Networks) to deliver that static content to your end users as well as provide additional resources on your Web and Application Servers to handle the extra load. To be prepared for that it is highly recommended to do some up-front large scale load-testing. For more information on that you can read my recent blog posts on To Load Test or Not to Load Test: That is not the Question.
Factor 2: Additional Content
The additional content in this case may have been necessary to fulfill the company’s marketing campaign. The question is whether the additional content could have been optimized to not download 40 additional resources with a total size of > 700kb.
Deep Dive Browser Diagnostics
Looking at the Dynatrace Browser Diagnostics Data we can observe several interesting aspects.
Observation 1: Load Impact of additional resources
We can see the actual impact of these additional resources that got downloaded during the time of the Super Bowl. Two of these resources took a very long time and with that had a major impact on page load time. The background image was delivered by their own web server and was not put on a CDN which results in more traffic on their web servers and not optimal performance for end users that are far away from their data centers. Another interesting aspect was an additional Facebook Like button that took 600ms to download and execute.
Observation 2: Page Transition Impact of CDN User Tracking Logic
When you are responsible for a website that is going to see a high user volume for a marketing campaign you want to
- Make sure the additional content for that marketing campaign is optimized, e.g: make sure this content is on CDNs and follow the Best Practices for Web Performance Optimization
- Make sure you test your application with the campaign specific features under realistic load
- Analyze the impact of 3rd party tracking tools and other widgets you put on your page