Live Blog: Tracking Super Bowl Ad Performance

Update: February 1st, 2015, 11:15pm EST

So let’s see during the game who was the fastest and who was the slowest in terms of response time.  In the following chart we see that Heinz, Cheerios, Beats, Subway and Carmax took the top spot.

top 5 2015-02-01 at 10.34.55 PM

Beats really surprised us, but lets come back to them in a moment.

Hear are slowest sites we say during the game. I think the standout was Pepsi who was very ambitious this year.  While their site showed as being one of the slowest, their design left us with some interesting questions as they appeared to load half of their page very quickly and then the rest of the page would take some time to load.  Interesting design decision.

bottom 5 2015-02-01 at 10.35.09 PM

But lets get back to Beats.  They really are a winner with their design this year.  They were able to push out 3 times more content (bytes) than Sonos and still be faster.  The key was keeping the number of connections and third parties to a minimum.

Screen Shot 2015-02-01 at 11.09.05 PM

Update: February 1st, 2015, 10:10pm EST

Bud Light struggles just after halftime.

Screen Shot 2015-02-01 at 10.10.56 PM

Again we to our Root Cause Analyzer for some insight and again the finger points at third parties.

Screen Shot 2015-02-01 at 10.11.55 PM

Interesting when we look at the errors we are seeing excessive number of redirect as the cause.

Screen Shot 2015-02-01 at 10.09.53 PM

Update: February 1st, 2015, 9:30pm EST

Teri Dalbec on our team has made a very interesting observation.  While not many sites are crashing completely we are seeing a substantial increase in the number of object errors on pages coming from third parties.

Here is a break down of errors by content type.


Here are some issues with objects coming from Facebook and Twitter


This is particularly bad on


What’s really alarming is that we are seeing this with javascript objects.  What functionality is missing because these javascripts aren’t being delivered?


Great observation from Teri Dalbec, thanks Teri!

Update: February 1st, 2015, 9:15pm EST

Looks like the pressure is starting to ease off.  Lots of sites that were red and yellow are turning green again.

Screen Shot 2015-02-01 at 9.08.51 PM

Update: February 1st, 2015, 8:55pm EST

Just to change things up, let’s have a look at our Sport Media Benchmark.  This is a benchmark  that is part of the Dynatrace Benchmarking practice ( .  We proactively track over 1500 web properties and then report our findings in a variety of industry specific categories.

Screen Shot 2015-02-01 at 8.53.12 PM

Here is an example of our Automotive Benchmark.  Notice that Dodge had a pretty substantial slow down during game time.

Screen Shot 2015-02-01 at 9.05.56 PM

Update: February 1st, 2015, 8:30pm EST

Katy Perry is crushing it during the half time show, however Katy Perry’s site is getting crushed at the same time.

Screen Shot 2015-02-01 at 8.22.44 PM

Looks like some issues on the server side.  Notice how calls to the php pages are having long First Byte Time issues.

Screen Shot 2015-02-01 at 8.22.31 PM

Monitoring PHP applications is easy with Dynatrace application monitoring.  While the screenshot below isn’t Katy Perry’s site, it does show how a PHP based application can be monitored on the server side.

Screen Shot 2015-02-02 at 9.30.41 AM

Update: February 1st, 2015, 8:10pm EST

Heading into half time let’s have a look at how we are seeing things shape up.  Here is the performance for the Advertisers for the past hour.

Screen Shot 2015-02-01 at 8.10.35 PM

Update: February 1st, 2015, 7:40pm EST

Crash the Super Bowl almost crashes…

crash 2015-02-01 at 7.35.43 PM

But manages to survive.

Screen Shot 2015-02-01 at 7.40.47 PM

Update: February 1st, 2015, 7:20pm EST

Looks like is having some issues. 

acocado 2015-02-01 at 7.15.01 PM

We are seeing socket connection errors and 500 errors.

errors 2015-02-01 at 7.18.45 PM

Update: February 1st, 2015, 6:55pm EST

Looks like BMW got hit as well.

bmw 2015-02-01 at 6.49.12 PM

Socket receive timeouts are usually an indicator of high traffic overwhelming network infrastructure.

bmw 2015-02-01 at 6.53.08 PM

Update: February 1st, 2015, 6:45pm EST

And even the opening act has issues.  Idina Menzel’s site gets crushed after she opens the Super Bowl.

idian 2015-02-01 at 6.39.23 PM

Update: February 1st, 2015, 6:30pm EST

Almost ready for Kickoff and many of the sites we are tracking are showing consistency issues. For example here we see by backbone location, some geographies are very consistent, some are having issues.

sonos 2015-02-01 at 6.21.26 PM

Update: February 1st, 2015, 5:40pm EST

So how are thing fairing on the Mobile front?  Here is a look of the First Byte Time coming from our Android tests.  This is how much time is taking the servers to respond to requests coming from Mobile Browsers.

FBT 2015-02-01 at 5.26.26 PM

Long First Byte Times indicate issues within the application tier for the advertiser.  While the screen shot below is not from one of the above advertisers it does show how application instrumentation with tools like Dynatrace application monitoring can provide complete end to end visibility across every tier down to methods within the application code to solve why there might be Long First Byte time.

txn flow map 2015-02-01 at 5.28.43 PM

Update: February 1st, 2015, 5:20pm EST

So here is an example of how third parties can cause a fumble (pardon the pun, I mean performance issues).  Here we see for the past two hours.

skittles 2015-02-01 at 5.15.43 PM

When we look at the detail we can see that the issue was due to some of the third parties their site calls having issues.

skittles 2015-02-01 at 5.16.12 PM

Update: February 1st, 2015, 5:10pm EST

As we mentioned, today we are looking at metrics other than just response time and availability.  Using the Superbowl Tumblr site we can see a view that puts some of the metrics into perspective.

metrics 2015-02-01 at 4.52.12 PM

The bottom two lines are DOM interactive followed by Time to First Paint.  These give us two indicators to tell us when the the page has loaded enough that javascript can start processing and interacting with page objects, and when the screen start to painting items.  The top two lines are DOM Complete and Total Page Load time.  This tells us when all of the javascript has loaded (not including asynchronous loads) and finished processing, and the Total Page Load time tells us when all of the assets have finished loading.

These pages can be incredibly complex, and what our numbers are showing us is that that those pages that are less complex tend to perform much better than those pages that are more complex.  Using indicators such as the numbers of hosts, objects and connections being made gives us an indication of how complex a page is.

Update: February 1st, 2015, 4:50pm EST

Let’s have a look at the fastest Advertisers from our Last Mile Network.

LM Fastest 2015-02-01 at 4.42.42 PM

Let’s look at the slowest Advertiser sites from our Last Mile Network.

LM slowest 2015-02-01 at 4.43.22 PM

Remember these results are for the past 24 hours from across the US from real end user machines.

Update: February 1st, 2015, 4:30pm EST

Snickers and M&M’s are back up.

snickers and mms 2015-02-01 at 4.34.20 PM

Update: February 1st, 2015, 4:15pm EST

Guess it makes sense since both are Mars properties, but Snickers is down as well.

snickers 2015-02-01 at 4.13.45 PM

Update: February 1st, 2015, 4:00pm EST

Looks like some issues starting with 2015-02-01 at 4.03.38 PM

Using our Root Cause Analysis we see the issue as a network level issue where socket connections are being refused.  This is being seen across our network from all locations. 2015-02-01 at 4.04.18 PM

I love M&Ms, hope they sort it out soon.

Update: February 1st, 2015, 4:00pm EST

The success and failure of many of the advertisers today will be based not on the infrastructure the host but based on the infrastructure of the third parties that will be providing them services.  Understanding how those third parties are performing and aggressively managing SLAs with those vendors can be make or break today.

Already our Outage Analyzer has caught some third party services having issues.  In this case we see an ongoing outage with (a service of that is impacting around 300 domains.

Outage Analyzer 2015-02-01 at 3.47.49 PM

Update: February 1st, 2015, 3:30pm EST

Is having some DNS related issues? We’ve been noticing some availability issues from 2015-02-01 at 3.14.33 PM

We can see some availability issues, but it is not showing 0% which means that their site is up, so something else must happening here.

When we look at the error detail, we can see that the issues are related to DNS failures.  The first thing we typically do is make sure that the DNS issue is not something being seen by other tests from the same locations.  We use Collective Intelligent Diagnostics to look at other tests running against other sites from the same locations, and that tells us that no one else is having DNS issues, so it is not a high level authority problem, nor a problem with the testing nodes.

Further diagnostics shows us the issue.

Kia Diagnostics 2015-02-01 at 3.10.44 PM

On the left is a DIG test being run from Verizon in Los Angeles, on the right is the same test being run from Level3 in Los Angeles.  In this case it looks like an issue at the CDN where local routing tables or caching instructions have not been updated.

This diagnostic capability is part of the power of using Synthetic testing, as Real User Monitoring would not show or discover this kind of issue.

Update: February 1st, 2015, 3:00pm EST

How are we tracking 100’s of tests, collecting 1000’s of data points, running across multiple networks?  Our team has created multiple dashboards that allow us to easily see when issues are occurring and correlate across these networks to understand what is causing these issues.

Below is an example of our Performance Dashboard, which provides a detailed view our how each advertiser is performing across the US from both our Backbone and Last Mile Networks.  The thresholds allow us to visually see when a particular geography is having a performance problems. It provides at glance details around performance, availability and errors.  This view of T-Mobile shows how we are seeing them across our Last Mile Network.

performance dashboard 2015-02-01 at 11.41.08 AM

Below is an example of our Operational Dashboard.  This provides a high level view of all the tests being run and provides it with simple red/yellow/green indicators when thresholds are being exceeded.  We see lots of red and yellow on this Dashboard because we’ve set our thresholds to be tight based on well-known industry standards.  We are using 4 seconds for a warning threshold and 8 seconds for a critical threshold.

operational dashboard 2015-02-01 at 11.42.18 AM

Below is our Third Party Dashboard.  This provides us with a view of every third party we discover when we load a page that we are testing.  It auto discovers new third parties that it hasn’t seen before and marks them with a star (for easy identification).   In this example, we see a Bud Light test which showed some new content/functionality being delivered from which hadn’t been seen before.  Since third parties can often call additional third parties, having this view is vital to understand exactly who is touching the advertiser’s end users.

third party dashboard 2015-02-01 at 11.43.52 AM

Update: February 1st, 2015, 2:30pm EST

We are not just tracking the advertisers but also some of the official Super Bowl sites.  For example, this year the NFL teamed up with NBC Sports and Tumblr to create a page to track ads.

Super Bowl NFL NBC Tumblr Page
Super Bowl NFL NBC Tumblr Page

It will be interesting to see how this partnership holds up.   Lots of moving parts and complexity in this page as seen with the Dynatrace AJAX Edition.  It takes 2.5 seconds before you see the First Impression and it took over 6 seconds to load.

Page Load Visualization of NFL NBC Tumblr Page using Dynatrace AJAX Edition
Page Load Visualization of NFL NBC Tumblr Page using Dynatrace AJAX Edition

Update: February 1st, 2015, 12:30pm EST

So it is past noon, lets start looking at some numbers from our Backbone Network.  Beats Music is coming out on top at 1.5 seconds and Jaguar is coming in last at 8.4 seconds.

Screen Shot 2015-02-01 at 12.32.59 PM

So why are we seeing so much of a difference between the fastest advertiser and slowest?  Beats music has 29 objects, makes 20 connections, uses 12 hosts and is delivering 6.5MB of data every time the page loads.  Jaguar has 196 objects, is making 97 connections, uses 71 hosts and is delivering 2.5MB of data…

WAIT A SEC…  How can this be? Beats is delivering over three time the amount of content and is still 4X faster than Jaguar.  It appears that brute forcing content through a CDN may not be as efficient as optimizing the site to make it less complex and use fewer third parties (hosts).

Update: February 1st, 2015, 11:30am EST

How to track performance during the Super Bowl? In the past we primarily relied on response time and availability. This year we will be using a number of different metrics to track advertiser performance. In addition to response time and availability this year we will be tracking the number of Third Parties (Hosts such as online ads providers, analytics, and social media), Connections and Objects (page complexity) and Byte Count (amount of data) each advertiser is using for their sites.

In addition we will be tracking “Time to First Paint”, this metric gives us an idea as to how responsive the page is without waiting for all of the content of the page to download.

In terms of response time, what we report on is the “Over the Wire” response time. We look at how much time it takes to download the complete page. This includes every page component (CSS, Javascript, Images, etc…) and takes into account things like http compression. While caching can improve performance, today we will be looking at the uncached view as most of the inbound traffic will not be returning visitors to these sites.

So from the above view we can see that the average number of connections across all of the advertisers is 48, average number of objects is 113, the average number of hosts is 26, the average number of bytes is 2.9MB, the average time to first paint is 1.015 seconds, and the average response time is 4.032 seconds.  We will compare individual advertisers to the average numbers for the group.

Update: February 1st, 2015, 10:00am EST

Super Bowl Sunday! After months of anticipation and preparation advertisers are finally ready to show their stuff during one of the most anticipated one day advertising events of the year.

As in the past, we at Dynatrace are tracking the major advertisers to see how the millions of page views their sites will generate today will impact their online presence.

Our tracking is done by executing tests from the Dynatrace Synthetic Monitoring Network. We are monitoring using a “9 Box” based approach over the US, where we test from 3 backbone locations on the east coast, 3 central locations and 3 west coast locations. This provides us with coverage across the US.   We will augment this testing with tests from our Last Mile Network, where tests are run from real end user machines.

This year we will be testing from our backbone network using our Internet Explorer agents as we want to track using some new metrics like “Time to First Paint”. We will be using Chrome agents from our last mile network as that remains the most popular browser.

Update: January 30th, 2015

This weekend we will be live blogging how different sites do once their ads air during the Super Bowl. Be sure to check back periodically to see our real-time analysis of how these sites are performing. Before the game, check out the predictions we made earlier this week. If you see a page loading slowly, let us know in the comments and we’ll check it out!

David Jones is the Director of Sales Engineering and AIOPs Evangelism for Dynatrace. He has been with Dynatrace for 10 years, and has 20 years’ experience working with web and mobile technologies from the first commercial HTML editor to the latest web delivery platforms and architectures. He has worked with scores of Fortune 500 organizations providing them the most recent industry best practices for web and mobile application delivery. Prior to Dynatrace he has worked at Gomez (Waltham), S1 Corp (Atlanta), Broadvision (Bay Area), Interleaf/Texcel (Waltham), i4i (Toronto) and SoftQuad (Toronto).