Dynatrace Performance Bowl 2017 Digital Experience, Live Blog Coverage

It’s that time of year again! The final game of the NFL season!  According to reports from Adweek 30-second spots are running $5 million USD. That is a substantial investment for a brief ad, and while everyone will gauge the success of the ads on their creativity, there will be an unspoken, hidden component as to whether these ads were truly success for the businesses advertising during the Big Game.


Final Results

That’s the game… Lets have a look at which advertisers won Dynatrace Performance Bowl. We actually have two categories, one for Desktop (Chrome browser) and one for Mobile (iPhone profile). For each category we are showing the results for the top 10 advertiser sites and an anonymous list of the bottom 10 performers.

Here are the results for Desktop (Chrome browser).

Congratulations to the team at P&G for www.mrclean.com.    Mr Clean was in a tough race with fractions of a second being the difference between the top spots.

Here are the results for Mobile (iPhone profile)

Congratulations again to the P&G team, but for a different page, this time febreze.com.

It’s been a wild ride tonight…   and now on to Perform 2017 


So Overtime in football means something completely different for Advertisers. For Advertisers there is no digital overtime. If something is overtime it is a time out, and time outs are great in football but bad for the web and mobile web. Here is an example of a page timing out from the Last Mile (this is from a real end user machine)

Fourth Quarter Update

If you are an advertiser do you know where data is coming from for your end users? Here is an interesting observation, for all of our monitoring this evening we are executing test from locations in the continental US. Below is a view of where we are monitoring Nintendo.com from within the US (who’s ad is premiering in the fourth quarter). So to be clear this is where our robot browsers are located.

Now below is a view of where the responses for our tests are coming from. Not the same picture is it? Not only is the picture different you can see that for tests originating in the US, response from Europe are being seen.

Understanding where responses are coming from is how advertisers can make sure that their routing is correct. If the Advertiser is using a CDN this is how synthetic testing can be used to help manage their CDN investment.

For the most part Advertisers have had excellent network conditions tonight, however we’ve noticed that Houston is is slowing down. When we looked into this a little more it looks like local network latency is the culprit.

Third Quarter Update

Lady Gaga rocked the half time show as did her site.

Advertisers thinking that they can turn to the Cloud and Containers to solve all of their server side problems take note. While the Cloud and Containers are a great opportunity to optimize and reduce costs they still need to be monitored, don’t let anyone tell you otherwise.

The big game is being held in Houston this year, Let’s have a quick look at how NRG Park’s Digital Experience is fairing?

84Lumber is starting to recover… good job guys turning that around quickly

Half Time Update

So far so good for most of the advertisers, however it has not been perfect for all of the ad providers. Dynatrace Synthetics is showing that for the past 24 hours we have executed 64 thousand tests and have seen a success rate of about 85%.

When a site is crushed by traffic during the big game, this is what we see…

In this case we are seeing what appear to be server side issues. This is unfortunate, but not unexpected.

Remember that on the server side, applications are very complex. This view is not of Lumber84, but is representative of how application communication components communicate.

This is a good news bad news situation… The good news is your creative created so much traffic demand you broke your servers. The bad news is, you broke your servers. I hope the Lumber84 team gets things back up soon.

Second Quarter Update

Lots of ads for movies so far, lots of social media interest in movie ads. Paramount came to the big game to win. Both Chrome Browser and Mobile Web performance for their sites are performing very well.

The Dynatrace Portal’s new Operational Monitoring view is making it much easier to track advertiser performance. It’s one thing to collect data, being able to analyze it and visualize is what makes the data useable.

When things go wrong, trying to diagnose the problem can be a painstakingly lengthy process. This is where machine learning algorithms (what people call AI) can help identify an issue in real time and dramatically reduce triage costs. For example, earlier today an advertiser we were tracking had an issue. In this case the Dynatrace synthetics test identified that an issue was occurring. There were dozens of page loads and hundreds of network requests which needed to be looked at. Below is a view of how the Dynatrace RCA (Root Cause Analysis) automatically identified the issue as coming from a third party.

The big game is being watched all across the globe, now would be a good time to talk a little bit about monitoring from a geographic perspective. This is a must as local routing from CDN (Content Delivery Networks) can be misrouted, or remote CDN PoPs (point of presence) where they serve data from can be oversubscribed. When looking at a traffic map like this when certain cities/regions start showing up red, you should be looking to see if your CDN is at fault.

First Quarter Update

Here is a quick view of some the advertisers who are performing well.

Coca-Cola started trending on twitter, their web site performance is rock solid.

Dynatrace Keynote Internet Health Report is showing no major internet routing/peering issues.

Dynatrace Outage Analyzer is showing that some tracking pixels and analytics tools were having issues prior to kickoff but have since recovered.

Mobile Advertiser Contributors

When we talk about complexity for Mobile Web Pages (from the client side) we have to look at the third parties contributing to the performance. One way to do this is to look at each individual domain, but to make things easier you can also group like kinds of contributors into groups. Below is a view of the contributor groups from multiple aggregated advertiser sites. By looking at the trend view we can see when one set of contributors is causing a performance issue. The other tiles show interesting views of data, for example CDN contribute the majority of data (byte count) but only represent 10% of the response time.

Last Minute Changes, Tuning & Tweaking

Even the best plans need some last minute tuning and tweaking. Below we can see some last minute DNS changes being made for the NFL’s big game page. DNS (Domain Name Server) is the essential internet “phone book” which resolves web site names to IP (network) addresses. Making sure DNS is performing is critical, and as we saw late last fall with the DYN DOS(Denial of Service) attack, DNS performance and availability can make or break the results for tonight advertisers.

90 minutes to go

Here is view of how we are tracking some of the advertiser web and mobile web sites.  We’ve divided the tests into web page tests as seen from a Chrome browser from known locations across the US (backbone locations).  We also have a category for tests running from an iPhone profile and from real user locations (Last Mile).   In addition we tracking contributor groups, these are third parties like ad networks, social media, analytics tools, etc…

Bounce… a fumble by any other name

Earlier we described how AI (Artificial Intelligence) can be used to manage real time performance and user behavior. Let’s talk about why these metrics can make or break the digital success for the brands advertising during this evenings big game.

Let’s talk about Bounce Rate. We define a bounce where a visitor enters the site/application and immediately leaves without performing any other action. Most digital business owners would agree that keeping the visitor on the site is better in all regards. Whether your conversion means more ad impressions (stickiness) or conversion is engagement or conversion is an online purchase… in any case, a bounce is counter to a successful conversion.

Below is an example of a Bounce Rate analysis. It shows the number of entry actions (where a visitor enters the site) sorted into buckets of performance (action duration) compared to the rate of Bounce. You can see that as actions take longer the higher the Bounce Rate becomes. This is a direct correlation on how performance impacts behavior.

Below the Bounce Rate analysis is the impact of JavaScript errors on bounce rate. JavaScript is code which runs in the browser. Often this code controls the functionality of the page, when there are JavaScript errors those errors can also impact the Bounce Rate. What makes JavaScript errors so problematic is that JavaScript’s can be introduced by third parties without the Game Advertiser even knowing it is there.

Whether the Game Advertiser pushes their web/mobile address in their ads or rely on SEO and search engines to push traffic to their sites, understanding where visitors are entering and exiting the site is critical. Below is an example of how Entry and Exit actions can be tracked in real time.

Knowing that a problem is occurring is the first step in reacting to it. This is why real time data is crucial. Below is some of the detail for a specific entry action. Note how when action duration (the site becomes slower) increases that the number of actions decrease and the bounce rate increases. This is a common occurrence.

Halftime… “I’m on the right track, baby”

So typically, the terms Digital Preparedness and Lady Gaga are not mentioned in the same sentence. But, here we are, blazing a new path. The organization that supports Lady Gaga’s digital properties know that they are an integral part of maintaining a world class media brand. As we’ve been describing in this blog, every web and mobile web property is running on a complex delivery chain. Here is an example of just how complex Lada Gaga’s page is when viewed in Google Chrome’s developer tools.

The network request overview (that’s the area in the middle with all the thin colored lines) shows all the requests being made when her page is loaded. These are exactly the requests our Synthetic monitoring looks at when loading a page as well.

Ahead of one of the most watched media events of the year we can see her team optimizing her website to improve perform and reduce costs.

How do we know this? Let’s have a look at some key web performance metrics we’ve seen by using some synthetic monitoring on her web and mobile web properties. Below is a view of a real time key metrics dashboard.

Each tile above serves a different purpose but are all derived from data from the same test.

The Response Time & Success Rate tile (top row left) shows that the success rate is 100%, very solid. More interestingly it shows that Saturday afternoon and change was implemented that improved response time by over 30% (going from an average of about 7 seconds to 4 seconds).

The Domain Contribution tile (top row 2nd from the left) shows how much time each domain (host) request adds to the overall response time. This makes it easy to understand if a third party (ad network, social media, analytics tool, etc…) is potentially creating a performance problem.

The Geographic Response Time tile (bottom row left) shows the response time from various cities across the US. These represent major end user population centers. This view makes it easy to see if a CDN (content delivery network) has misrouted traffic or an oversubscribed point of presence. CDN’s represent a major investment which must be managed.

The Key Delivery Indicator tile (bottom row 2nd from the left) shows the delivery characteristics of the properties (pages) being monitored. This tile shows byte count (the amount of data being delivered), object count (the number of network resource requests being made), connection count (the complexity of the page) and host count (the number of third parties being used). Here you can see that Lady Gaga’s site has been tweaked for performance as these values have all been reduced. Optimizing these characteristics improves performance and reduces costs (less bandwidth consumed, few resource requests mean that hardware can support more traffic, etc…).

The four other tiles track critical metrics for DNS Health, Network Latency, Server Performance and Client Side Performance. We have seen recent examples like the DYN attack [insert link] in the fall and Cyberweek issues [insert link] which show that ignoring these metrics can be like ignoring a blitz when you are 3rd and nine with two minutes to go.

There are other big games today.

The “Big Game” is not the only big game today. Don’t forget the Animal Planet Puppy Bowl, we haven’t.  We’ll be watching the web performance of the Puppy Bowl as well as Lady Gaga’s site.  More on Lady’s gaga’s site in a bit.

The big game is only hours away.

Only hours to go and you can feel the anticipation in the air. This year we are live blogging from the Cosmopolitan Hotel in Las Vegas a day before Perform 2017 starts. It is a fortunate coincidence that the big game is happening right before one of the premier conferences on digital transformation management. We are going to see real world examples later today of brands digital platforms performing brilliantly along with examples of missed digital field goal attempts (I’m old enough to remember watching the Bills go to the show 4 times in a row… I’m looking at you Norwood).

We are tracking over 50 major brands and have over 150 tests and by the end of the day we will have looked at thousands of page and mobile page loads and tens of thousands of server requests. It is a lot of data to look at, and therein lies the rub of monitoring digital applications… the amount of data which gets generated. At a certain point, it becomes unmanageable for an individual operator to watch everything. We won’t be alone with this challenge, the brands advertising during today’s big game have similar challenges… ridiculously complex applications generating massive amounts of real time data.

When tracking the end user performance of an event (this is the most important point of view) it is helpful to look at the data from two perspectives: Performance and Behavior. Performance is based on the amount and types of traffic hitting your site, along with the number of and duration of user actions (clicks, touches, swipes, page loads, etc…). Behavior is based on the number and types of sessions, where users enter and exit the site, how many bounces and conversions are seen.

So how do these brands manage this for every visitor? How do they understand every action? How do they do this in real time? The answer is that they have to rely on machine learning, algorithms, what people commonly call AI (Artificial Intelligence).

Below is an example of Dynatrace’s Performance Analysis display. Here we see real time analysis of gap free data (every visit and every action) to automatically identify performance problems and their root cause.

For the behavior perspective, the display below show User Behavior analysis. Here we see real time analysis of new vs returning user behavior, Entry actions and exit actions. We see how many visitors are bouncing off the site and overall conversion success. Again that same AI allows us to identify behavior issues and problems in real time and point to the root cause if it is a performance problem.

These two perspectives show how useful AI can be used for managing a digital event like the big game. They key is that every visit counts, to make this work best you should use it on 100% of the traffic coming to the digital property.

Keep this blog bookmarked this is just the first of many updates we will be making today.

Why are advertiser sites at risk during events like the big game?, Feb 4

The challenge, as we highlighted yesterday, is that the advertiser’s applications supporting their ad activity during and after the big game are exceptionally complex and getting more complex every year.

For example, below is a view from a luxury brand of what their application looks like from the server side (behind their firewall).  We can see multiple services and processes all talking with – and depending on – each other.  In many cases these services share the same host (you can think of a host as a machine) and compete for the same (limited) resources (CPU, memory, disk, network, etc.).  At any point, any of these dependencies could be at risk and cause an issue that could impact other compute resources that touch an end user.

Advertisers need to understand and embrace this complexity. There is no single “server” to point the finger at any more.  Compute resources can be elastic and created and destroyed on demand depending on how much load is hitting their applications.

What touches the end user exists on the end of a complex delivery chain, in which everything needs to work perfectly every time, or the advertiser risks a brand-damaging experience during and after the big game.

Advertisers need to remember that anything along this digital delivery chain can become a potential point of failure, from DNS failing to a CDN misrouting traffic, to a third party causing a javascript to fail, to code running in the cloud failing, to infrastructure being under sized, etc…

Advertisers that embrace Digital Performance Management understand that managing this complexity is the IT challenge that keeps them from being competitive and prevents them from innovating.

The Dynatrace Team loves watching Pro Sports performance

The team here at Dynatrace loves watching pro sports performance, and we aren’t talking about just watching Monday night football.  For the past year, our team has been watching the digital performance of major sports websites.  Pro sports may have at one time been dominated by traditional media channels, but the digital transformation we see in every industry has hit Pro Sports hard, like a linebacker sacking a quarterback.  The pro sports industry has been attempting to get a grip on what digital transformation means to them, is it mechanism to push more eyes to TV screens? Is it a medium to stream content to viewers on mobile devices? Is it a means by which they can generate more ad revenue?  The answer to all of these questions is… yes, and more.

Digital transformation is redefining how pro sports consumers interact with these brands.  If that digital transformation does not perform well, or has issues it damages that brand.  We are often asked… so what if some thing is slow, why should my business care?  The answer to this is simple, everything is better when it performs well.  If a consumer has a choice to watch streamed content on a high bandwidth or slow bandwidth connection, which will they choose?  If a consumer has a choice to go to a fast loading mobile web site or a slow loading mobile web site, which will they choose?  If a business has a choice to run slow performing code or fast performing code in the cloud, which will they choose?

The answers to the above questions may seem obvious, high bandwidth streams provide better user experience, a fast loading mobile site leads to better user experience and higher conversion rates, fast performing code reduces cloud compute resources and costs.  How you make a mobile page load faster or make code run faster is actually pretty tough without the right data.

Below is a one year trend view of major sporting brands main web properties.  This data was taken from a real browser from know locations under known conditions.

You can see changes in performance as the sites themselves change.  In some cases that change leads to improved performance, in some cases the performance got worse.  The applications which support and deliver these sites are incredibly complex, a small change in how the site is delivered can lead to performance improvement or performance slowing down.   Anything from changing content on the front end, code on the back end, infrastructure running the code can impact how the application performs and how end users perceive their experience.

After watching the performance of so many brands and working with so many businesses, what the team at Dynatrace knows is that the complexity in managing these applications is increasing at an alarming rate.  The challenge for these brands is how to manage this change.  Traditional application performance management tools are falling short in providing these brands with what they needs to ensure quality end user experience.

We will talk more in next couple of days about how a Digital Performance Management platform is what these brands need to address the increasing complexity of their digital transformation.

Don’t let digital complexity fumble your 3rd and goal opportunity.

It’s that time of year again! The final game of the NFL season!  According to reports from Adweek 30-second spots are running $5 million USD. That is a substantial investment for a brief ad, and while everyone will gauge the success of the ads on their creativity, there will be an unspoken, hidden component as to whether these ads were truly success for the businesses advertising during the Big Game. That hidden component will be entirely based on the success of each advertiser’s ability to handle the digital demand generated from the ads, and deliver a strong digital experience throughout the event.   The advertisers need to make sure they do not squander their 3rd-and-goal position by having their digital properties fumble at the goal line because they are unable to support the traffic generated by their Big Game ad.

These advertisers represent some of the most well-known brands, and the $5 million cost of entry for an ad is only the tip of the iceberg for the costs associated with creating and maintaining the brand’s relationships with its customers. While any number of elements can result in a poor digital experience on a web or mobile site, the most difficult element will be managing the complexity of the advertiser’s digital channels. Web and Mobile channels have been becoming increasingly complex, and anything along the digital delivery chain can become a point of failure that causes the entire digital experience to get sacked before crossing into the end zone.

As we have in the past, the team at Dynatrace will be watching and reporting on the web and mobile performance for the advertisers during the Big Game with our live blog coverage.  The Dynatrace “Performance Bowl” will be tracking the following:

  • NFL websites
  • Ticket websites
  • Team websites
  • Betting/Gaming/Fantasy Football websites
  • Official Broadcaster website
  • Major sports sites

And of course:

  • Advertiser websites

We will be tracking these proactively using real browsers running from multiple locations across the US and the globe.

As we did last year (/blog/prepping-for-the-big-game-2016/) we will be providing a live blog before, during, and after the game. Keep checking in to see how the sites we are tracking are performing.

Super Bowl , SuperBowl

David Jones is the Director of Sales Engineering and AIOPs Evangelism for Dynatrace. He has been with Dynatrace for 10 years, and has 20 years’ experience working with web and mobile technologies from the first commercial HTML editor to the latest web delivery platforms and architectures. He has worked with scores of Fortune 500 organizations providing them the most recent industry best practices for web and mobile application delivery. Prior to Dynatrace he has worked at Gomez (Waltham), S1 Corp (Atlanta), Broadvision (Bay Area), Interleaf/Texcel (Waltham), i4i (Toronto) and SoftQuad (Toronto).