Update: February 1st, 2015, 11:15pm EST
So let’s see during the game who was the fastest and who was the slowest in terms of response time. In the following chart we see that Heinz, Cheerios, Beats, Subway and Carmax took the top spot.
Beats really surprised us, but lets come back to them in a moment.
Hear are slowest sites we say during the game. I think the standout was Pepsi who was very ambitious this year. While their site showed as being one of the slowest, their design left us with some interesting questions as they appeared to load half of their page very quickly and then the rest of the page would take some time to load. Interesting design decision.
But lets get back to Beats. They really are a winner with their design this year. They were able to push out 3 times more content (bytes) than Sonos and still be faster. The key was keeping the number of connections and third parties to a minimum.
Update: February 1st, 2015, 10:10pm EST
Bud Light struggles just after halftime.
Again we to our Root Cause Analyzer for some insight and again the finger points at third parties.
Interesting when we look at the errors we are seeing excessive number of redirect as the cause.
Update: February 1st, 2015, 9:30pm EST
Teri Dalbec on our team has made a very interesting observation. While not many sites are crashing completely we are seeing a substantial increase in the number of object errors on pages coming from third parties.
Here is a break down of errors by content type.
Here are some issues with objects coming from Facebook and Twitter
This is particularly bad on http://www.dovemencare.us
Great observation from Teri Dalbec, thanks Teri!
Update: February 1st, 2015, 9:15pm EST
Looks like the pressure is starting to ease off. Lots of sites that were red and yellow are turning green again.
Update: February 1st, 2015, 8:55pm EST
Just to change things up, let’s have a look at our Sport Media Benchmark. This is a benchmark that is part of the Dynatrace Benchmarking practice (http://www.dynatrace.com/benchmarks/) . We proactively track over 1500 web properties and then report our findings in a variety of industry specific categories.
Here is an example of our Automotive Benchmark. Notice that Dodge had a pretty substantial slow down during game time.
Update: February 1st, 2015, 8:30pm EST
Katy Perry is crushing it during the half time show, however Katy Perry’s site is getting crushed at the same time.
Looks like some issues on the server side. Notice how calls to the php pages are having long First Byte Time issues.
Monitoring PHP applications is easy with Dynatrace application monitoring. While the screenshot below isn’t Katy Perry’s site, it does show how a PHP based application can be monitored on the server side.
Update: February 1st, 2015, 8:10pm EST
Heading into half time let’s have a look at how we are seeing things shape up. Here is the performance for the Advertisers for the past hour.
Update: February 1st, 2015, 7:40pm EST
Crash the Super Bowl almost crashes…
But manages to survive.
Update: February 1st, 2015, 7:20pm EST
Looks like avocadosfrommexico.com is having some issues.
We are seeing socket connection errors and 500 errors.
Update: February 1st, 2015, 6:55pm EST
Looks like BMW got hit as well.
Socket receive timeouts are usually an indicator of high traffic overwhelming network infrastructure.
Update: February 1st, 2015, 6:45pm EST
And even the opening act has issues. Idina Menzel’s site gets crushed after she opens the Super Bowl.
Update: February 1st, 2015, 6:30pm EST
Almost ready for Kickoff and many of the sites we are tracking are showing consistency issues. For example here we see Sonos.com by backbone location, some geographies are very consistent, some are having issues.
Update: February 1st, 2015, 5:40pm EST
So how are thing fairing on the Mobile front? Here is a look of the First Byte Time coming from our Android tests. This is how much time is taking the servers to respond to requests coming from Mobile Browsers.
Long First Byte Times indicate issues within the application tier for the advertiser. While the screen shot below is not from one of the above advertisers it does show how application instrumentation with tools like Dynatrace application monitoring can provide complete end to end visibility across every tier down to methods within the application code to solve why there might be Long First Byte time.
Update: February 1st, 2015, 5:20pm EST
So here is an example of how third parties can cause a fumble (pardon the pun, I mean performance issues). Here we see Skittles.com for the past two hours.
When we look at the detail we can see that the issue was due to some of the third parties their site calls having issues.
Update: February 1st, 2015, 5:10pm EST
As we mentioned, today we are looking at metrics other than just response time and availability. Using the Superbowl Tumblr site we can see a view that puts some of the metrics into perspective.
These pages can be incredibly complex, and what our numbers are showing us is that that those pages that are less complex tend to perform much better than those pages that are more complex. Using indicators such as the numbers of hosts, objects and connections being made gives us an indication of how complex a page is.
Update: February 1st, 2015, 4:50pm EST
Let’s have a look at the fastest Advertisers from our Last Mile Network.
Let’s look at the slowest Advertiser sites from our Last Mile Network.
Remember these results are for the past 24 hours from across the US from real end user machines.
Update: February 1st, 2015, 4:30pm EST
Snickers and M&M’s are back up.
Update: February 1st, 2015, 4:15pm EST
Guess it makes sense since both are Mars properties, but Snickers is down as well.
Update: February 1st, 2015, 4:00pm EST
Looks like some issues starting with www.mms.com
Using our Root Cause Analysis we see the issue as a network level issue where socket connections are being refused. This is being seen across our network from all locations.
I love M&Ms, hope they sort it out soon.
Update: February 1st, 2015, 4:00pm EST
The success and failure of many of the advertisers today will be based not on the infrastructure the host but based on the infrastructure of the third parties that will be providing them services. Understanding how those third parties are performing and aggressively managing SLAs with those vendors can be make or break today.
Already our Outage Analyzer has caught some third party services having issues. In this case we see an ongoing outage with sync.tidaltv.com (a service of videologygroup.com) that is impacting around 300 domains.
Update: February 1st, 2015, 3:30pm EST
Is Kia.com having some DNS related issues? We’ve been noticing some availability issues from Kia.com
We can see some availability issues, but it is not showing 0% which means that their site is up, so something else must happening here.
When we look at the error detail, we can see that the issues are related to DNS failures. The first thing we typically do is make sure that the DNS issue is not something being seen by other tests from the same locations. We use Collective Intelligent Diagnostics to look at other tests running against other sites from the same locations, and that tells us that no one else is having DNS issues, so it is not a high level authority problem, nor a problem with the testing nodes.
Further diagnostics shows us the issue.
On the left is a DIG test being run from Verizon in Los Angeles, on the right is the same test being run from Level3 in Los Angeles. In this case it looks like an issue at the CDN where local routing tables or caching instructions have not been updated.
This diagnostic capability is part of the power of using Synthetic testing, as Real User Monitoring would not show or discover this kind of issue.
Update: February 1st, 2015, 3:00pm EST
How are we tracking 100’s of tests, collecting 1000’s of data points, running across multiple networks? Our team has created multiple dashboards that allow us to easily see when issues are occurring and correlate across these networks to understand what is causing these issues.
Below is an example of our Performance Dashboard, which provides a detailed view our how each advertiser is performing across the US from both our Backbone and Last Mile Networks. The thresholds allow us to visually see when a particular geography is having a performance problems. It provides at glance details around performance, availability and errors. This view of T-Mobile shows how we are seeing them across our Last Mile Network.
Below is an example of our Operational Dashboard. This provides a high level view of all the tests being run and provides it with simple red/yellow/green indicators when thresholds are being exceeded. We see lots of red and yellow on this Dashboard because we’ve set our thresholds to be tight based on well-known industry standards. We are using 4 seconds for a warning threshold and 8 seconds for a critical threshold.
Below is our Third Party Dashboard. This provides us with a view of every third party we discover when we load a page that we are testing. It auto discovers new third parties that it hasn’t seen before and marks them with a star (for easy identification). In this example, we see a Bud Light test which showed some new content/functionality being delivered from Cloudfront.net which hadn’t been seen before. Since third parties can often call additional third parties, having this view is vital to understand exactly who is touching the advertiser’s end users.
Update: February 1st, 2015, 2:30pm EST
We are not just tracking the advertisers but also some of the official Super Bowl sites. For example, this year the NFL teamed up with NBC Sports and Tumblr to create a page to track ads.
It will be interesting to see how this partnership holds up. Lots of moving parts and complexity in this page as seen with the Dynatrace AJAX Edition. It takes 2.5 seconds before you see the First Impression and it took over 6 seconds to load.
Update: February 1st, 2015, 12:30pm EST
So it is past noon, lets start looking at some numbers from our Backbone Network. Beats Music is coming out on top at 1.5 seconds and Jaguar is coming in last at 8.4 seconds.
So why are we seeing so much of a difference between the fastest advertiser and slowest? Beats music has 29 objects, makes 20 connections, uses 12 hosts and is delivering 6.5MB of data every time the page loads. Jaguar has 196 objects, is making 97 connections, uses 71 hosts and is delivering 2.5MB of data…
WAIT A SEC… How can this be? Beats is delivering over three time the amount of content and is still 4X faster than Jaguar. It appears that brute forcing content through a CDN may not be as efficient as optimizing the site to make it less complex and use fewer third parties (hosts).
Update: February 1st, 2015, 11:30am EST
How to track performance during the Super Bowl? In the past we primarily relied on response time and availability. This year we will be using a number of different metrics to track advertiser performance. In addition to response time and availability this year we will be tracking the number of Third Parties (Hosts such as online ads providers, analytics, and social media), Connections and Objects (page complexity) and Byte Count (amount of data) each advertiser is using for their sites.
In addition we will be tracking “Time to First Paint”, this metric gives us an idea as to how responsive the page is without waiting for all of the content of the page to download.
So from the above view we can see that the average number of connections across all of the advertisers is 48, average number of objects is 113, the average number of hosts is 26, the average number of bytes is 2.9MB, the average time to first paint is 1.015 seconds, and the average response time is 4.032 seconds. We will compare individual advertisers to the average numbers for the group.
Update: February 1st, 2015, 10:00am EST
Super Bowl Sunday! After months of anticipation and preparation advertisers are finally ready to show their stuff during one of the most anticipated one day advertising events of the year.
As in the past, we at Dynatrace are tracking the major advertisers to see how the millions of page views their sites will generate today will impact their online presence.
Our tracking is done by executing tests from the Dynatrace Synthetic Monitoring Network. We are monitoring using a “9 Box” based approach over the US, where we test from 3 backbone locations on the east coast, 3 central locations and 3 west coast locations. This provides us with coverage across the US. We will augment this testing with tests from our Last Mile Network, where tests are run from real end user machines.
This year we will be testing from our backbone network using our Internet Explorer agents as we want to track using some new metrics like “Time to First Paint”. We will be using Chrome agents from our last mile network as that remains the most popular browser.
Update: January 30th, 2015
This weekend we will be live blogging how different sites do once their ads air during the Super Bowl. Be sure to check back periodically to see our real-time analysis of how these sites are performing. Before the game, check out the predictions we made earlier this week. If you see a page loading slowly, let us know in the comments and we’ll check it out!