Luxury car makers Audi and Mercedes took differing approaches to generating brand awareness during the 2013 Super Bowl: Audi, through its “Prom” big game commercial and subsequent #BraveryWins hash tag campaign on Twitter; Mercedes via its “Soul” big game commercial, featuring a few high-profile celebrities and a very aggressive price point, appealing to a younger demographic.
Both companies also took advantage of a “second-screen” strategy by linking the big game commercials to their websites, with the intent of driving web traffic following their respective ads airing. We collected performance data from both the Backbone and Last Mile of the Internet through the Compuware APM SaaS network between 5pm and 11pm on Super Bowl Sunday, which gave us some insight into the successes that Audi and Mercedes had in handling the increased traffic from their campaigns:
- Audi maintained a relatively stable average response time of 3.487 seconds from the Backbone, with the Last Mile averaging 12.042 seconds
- Mercedes maintained an approximate 400 millisecond edge on Audi on average from the backbone (3.060 seconds), and a 2 second edge from the Last Mile (10.227 seconds), yet experienced large spikes in response times during the event
- From a geographic perspective, the worst areas hit appeared to be on the West Coast for Audi (specifically San Jose and Seattle with average response times > 4 seconds), and the US Central region for Mercedes (specifically Chicago, Dallas, and Kansas City with average response times > 4 seconds)
So, who won the battle of web performance between these two companies during Super Bowl Sunday? Was it Audi with its consistent “slow and steady” approach or Mercedes with its high-risk, high-reward “speed racer” approach?
Backbone Analysis – Operational Focus
As described earlier, both Audi and Mercedes maintained stable average response times, starting with the pre-game show and through the Super Bowl event.
Shortly after the Pepsi Halftime show, Mercedes began to lose its edge, with Backbone data trending higher as buzz for the upcoming ad grew. Ironically enough, this was about the same time that the infamous 3rd quarter power outage occurred. In the spirit of rivalry, Audi took to Twitter during the power outage and offered a jab at Mercedes, who owns corporate naming rights to the Superdome (with their logo being prominently displayed every time the cameras panned across the top of the stadium).
Right around 9:30pm when this tweet was sent, Mercedes’ home page www.mbusa.com began to experience multiple failures: Gateway Timeouts, Socket Receive Timeouts, and Connection Timeouts at the object level resulted in Wait for Page Complete Timeouts at the script level, causing the tests to fail intermittently over the next 2 hours. This pattern of object and page level errors was repeated and escalated following the airing of the Mercedes ad during the 4th quarter, yielding yet another spike in average response times.
Despite these spikes and errors, Mercedes still maintained faster response times on average over the Super Bowl Sunday event window (3.060 seconds for Mercedes vs. 3.487 seconds for Audi). So, how could the Mercedes site continue to appear faster than Audi, even with all of these issues? To answer this, 3 key questions need to be addressed:
1) Where were objects being downloaded from, and how many were there?
2) What caused the most performance issues: internal hosts or 3rd parties?
3) Which web strategy offers the best repeatable strategy and the most efficient performance for my website?
1) Host Domain Analysis
Mercedes utilized a distributed web strategy, leveraging a major CDN to push out a majority of page content from the www.mbusa.com internal host domain. Audi also utilized a distributed web strategy, leveraging two major CDN providers to allocate content from its www.audiusa.com and progress.audi.com internal domains. Mercedes found issues in connection times to the CDN hosting the www.mbusa.com host domain at around 9:30pm, subsequently increasing object response times due to increased network latency. This resulted in degradation of web request execution, increasing Mercedes’ average response times both before and during its ad airing.
Best Practice Tip: Having access to a measurement tool or services that provides deep visibility into host/IP address detail at the connection level allows you to quickly identify the source of performance issues, no matter if they are the result of issues with internal or 3rd party hosts. This perspective also provides organizations with the ability to clearly see how page objects are distributed across the hosts so that they can work to find ways to optimize this distribution.
Component Level Breakdown
Throughout the Super Bowl time period, Mercedes experienced large spikes in the 1st byte measurement component before and after the airing of the big game commercial, as well as increased connection times to the www.mbusa.com host domain. We use the 1st byte time to describe the time required for a web server to send the first byte of data back to the browser in response to a web request. It serves as an indicator of application layer health for web applications.
Spikes in 1st byte time with increased connection times for the Mercedes site confirm that the CDN infrastructure supporting the primary Mercedes host – www.mbusa.com – experienced challenges in responding to heavier web traffic in a timely fashion. In contrast, Audi maintained stable 1st byte times during the Super Bowl, indicating that the distributed web strategy used by Audi was more effective in handling the increased traffic.
Best Practice Tip: Without visibility into component detail at the measurement and object level, it can be more difficult to validate the possible causes of issues identified via host level analysis, increasing the mean time to resolution, with the potential for revenue loss and brand damage effects to follow.
Last Mile Analysis – End User Focus
Analyzing measurement results from the Last Mile provides a synthetic perspective on performance from real world, end-user connections captured from consumer laptops/desktops.
Looking at the Last Mile data, spikes are clearly visible for both Audi and Mercedes during the Super Bowl Sunday event window, seemingly running counter to what was discussed above. This is often the case as the true network effects of large performance events affect the everyday internet users to a far greater degree than can ever be detected from the perspective of a Backbone measurement in a datacenter.
Audi’s major spike occurred between 6 and 7pm EST, around the time when its ad aired during the 1st quarter, with Mercedes encountering multiple performance spikes throughout the collection period, the largest occurring around the time of the power outage at 9:30pm. Both Audi and Mercedes experienced variable Last Mile results, but overall end users using their desktop computers with broadband connections to the Internet were experiencing page load times from Mercedes that were approximately 2 seconds faster than those of Audi.
Best Practice Tip: When determining the effectiveness of your CDN strategy, it is crucial to validate that performance from Last Mile data aligns with Backbone results as one step in verifying and validating the effectiveness of the investment.
2) Contributor Group Analysis
Looking at the contributor groups (a mapping of host names and IP addresses identified as either internal or 3rd party), measurement data identified that the majority of data was being downloaded from only a small number of domains: progress.audi.com and www.audiusa.com for Audi; www.mbusa.com for Mercedes. Mercedes, from an internal contributor group perspective, yielded average response times of 37.833 seconds. Audi, from an internal contributor group perspective, maintained a steady average response time 18.186 seconds. Although average response times for both Audi and Mercedes were much lower than these, the contributor group perspective sheds light on the role those specific hosts played in affecting overall end-user experience.
Best Practice Tip: It is imperative to have an understanding of how 3rd party or internal host domains affect your end-users. When deploying a CDN strategy, visibility at the Last Mile data can help to identify hot spots (number of bytes downloaded and average response times from contributor groups), in order to optimize performance from an end-user perspective.
3) Page Composition Breakdown
From a page composition perspective, Mercedes maintained an average page weight of 1.7MB, with an average of 76 page objects and an average of 1.49MB worth of the total page content contributed by the www.mbusa.com internal host. Audi maintained an average page weight of 2.1MB, with an average of 114 page objects, with an average of 1.05MB and 530KB worth of page content contributed by the www.audiusa.com and progress.audi.com internal hosts respectively. Audi’s larger page weight and total number of page objects limited their ability to beat Mercedes’ overall response times despite more consistent performance.
Best Practice Tip: Visibility into page composition from the perspective of the end-user can frame how performance is affected by page size and number of objects. This can assist in directing site optimization efforts, such as adding CSS sprites, off-loading more page objects to supporting web services, or content compression.
So, who won the battle of web performance, and what are the lessons learned?
For websites with high volumes of dynamic content, Audi’s web performance strategy can enable you to maintain stable and predictable performance through periods of heavy load traffic. In addition, streamlining the page in reference to number of objects and subsequent page weight can decrease overall average response times as was shown with Mercedes results before and after the Super Bowl. Furthermore, a keen understanding of how CDN’s and other 3rd party providers can impact your performance strategy is vital to maximizing end-user experience. Given the complexity of the Internet, internal data center infrastructures, as well as application architectures across the entire delivery chain, which performance strategy would you choose?
To learn more best practices for integrating 3rd party content, we recommend reading, ‘Third Party Content Management applied: Four steps to gain control of your Page Load Performance!’.