Update: December 9, 2015 at 9:00am EST

The Mobile Holiday Weekend

Now that the dust has settled let’s have a look at how mobile impacted the Holiday weekend. Dynatrace’s holiday shopping report told us that 50% of millennials will do more mobile shopping than they will by making in store purchases, and 60% of them will be doing more mobile shopping than they did last year. With that in mind, let’s look at some of the numbers we saw during the Holiday weekend.

The following numbers came from the Dynatrace Mobile Retail Benchmark. This benchmark tests mobile sites using an iPhone device profile, driving a mobile browser, across real wireless networks (AT&T, Sprint and Verizon) from Boston, Chicago and Santa Clara. These results are averages from Nov 25th through Dec 1st.

Update_1

These results are the top 10 from a list of 50 online U.S. retailers. Costco, Tiger Direct, Amway, Apple, Overstock and Walgreens were all faster than 3 seconds. Why is that important? Our holiday shopping report told us that 49% of shoppers would abandon a mobile site if it took more than 3 seconds to load.

When we look at these results many retailers ask how they can improve their performance. One of the suggestions we have is to compare your performance with other retailers to see how they are delivering their mobile websites.

The table below shows how the top 10 U.S. mobile performers are delivering their sites compared to the bottom 10.  The general trend is that sites that really optimize their responsive design to decrease page weight, which become less complex in terms of object and connection requests and reduce the number of third parties, will perform better.

Update_2

Our holiday shopping report told us that 62% of millennials will use their mobile phones when in brick and mortar stores and 37% would use their mobile devices while in physical stores to make purchases.

In addition to the U.S., Black Friday and Cyber Monday shopping events are making their way into other countries.  Here is a look at retailers from the U.K. whose mobile web sites loaded in under 3 seconds.

Update_3

The U.K. had a lot of mobile web sites which performed under 3 seconds, we believe that this has a lot to do do with the maturity of the U.K. mobile marketplace.  Here are some results from French retailers whose mobile web sites performed under 3 seconds.

Update_4

It is clear that Black Friday and Cyber Monday is becoming a global event, retailers from across the globe will need to start planning and preparing for these sales events in ways that they haven’t in the past.

When we look at some real traffic from an online retailer for the Holiday week, we can see some interesting patterns. Below the dark blue bar represents Mobile Safari traffic.  The time of day definitely impacts what devices shoppers use to access retail sites.

Update_5

When we look at the user satisfaction view below we can see that users are definitely using different versions of Mobile Safari, and that different versions can provide different experiences.  Retailer need to keep this in mind and not tailor sites to specific versions of a mobile device.  Retailers should be making their applications as inclusive as possible.

Update_6

A couple of visits to this retailer were actually showing up as a Tesla car browser…that really is mobile!

Understanding how mobile web applications behave is key for retailers to ensure future success. Retailers will never control what devices shoppers use so they need to adapt to trends in mobile devices quickly. Given the sheer variety of devices in the market, retailers will need to automatically become aware of issues with specific devices. Below is an example of a next generation Dynatrace solution (Ruxit) that will automatically discover when a particular mobile device is causing an issue, and what is causing that issue.

Update_7

If retailers intend on remaining competitive they will need to adaptive to the ever changing mobile shopper.

Update:November 30th, 2015 at 11:50pm EST

It’s been a busy Holiday Weekend, there has been a lot of success and a couple of slip ups.  We’ve been focusing on response times and details making up those response times, but the key thing to remember is that Retailers need to focus on the digital experience of their customers.  One thing that changed this year is the increase in how mobile is impacting retailers.  Our Holiday Survey this year puts this into perspective….

50% of Millennials will do more shopping on their smartphones and tablets than they will by making in store purchases.

60% of Millennials will be doing more mobile shopping than last year.

62% of Millennials use mobile devices in physicals stores for price comparison, product reviews and coupons

37% of Millennials use mobile devices in physical stores to make purchases

Regardless of whether the customer was using traditional web sites, or mobile web sites/applications, consumers demand an exceptional digital experience.  Our Holiday Survey found…

81% of Millennials will abandon buggy sites/apps and shop elsewhere

51% of Millennials will complain on social media

49% of Millennials abandon sites/apps if they fails to load in 3 seconds

Update: November 30th, 2015 at 11:30pm EST

Having another look at Outage Analyzer.  For Cyber Monday, Outage Analyzer saw hundreds of 3rd party outages, impacting (potentially) 52,024 domains websites with over 308,329 affected website hours when we total up the outages.

Outage_Analyzer_113015pm

Retailers need to understand the impact that third parties have on end user experience. There needs to be a valid reason to include a 3rd party and Retailers should audit their site on a regular basis.  We’ve had discussions with Retailers about their 3rd parties and on several occasion after an audit was completed over 50% of the 3rd parties on their site was no longer being used by any part of the business.

Update: November 30th, 2015 at 11:00pm EST

Having a look at EMEA Benchmark results.  German and Italian Retailers struggled Monday evening as response times slowed for many sites.

EMEA_Benchmarks_CyberMonday

Update: November 30th, 2015 at 5:00pm EST

Cyber Monday came on like gang busters living up to its name.  Here is our batch of numbers looking at the past 24 days for Cyber Monday (We switched from the past 5 days just to show the churn and volatility we are seeing today).  We will start by looking at the Retail Home Page Benchmarks.  The companies on this chart are providing the Top 10 fastest retail home pages.  Apple, REI and Staples still taking the top 3 spots.

Top10_HP_pm_113015

The average across the Retail Home Page Benchmark slowed to 4.9 seconds, and the slowest site took over 10 seconds to open.

In this chart we are looking at the top 10 retailers for online transactions, where we go to the home page, search for a product, look at the details, add it to a shopping cart and check out.  Apple, Tiger Direct and Fanatics take the top three spots.

Top10_TXN_pm_113015

The average across the Product Order Transaction Benchmark slowed to 30.2 seconds, and the slowest retailer was over 63 seconds to complete a checkout transaction.

Finally in this chart we are looking at the mobile performance for online retailers.  These tests are executed from real wireless carriers from across the US.  The fastest mobile web retailers are Costco, Tiger Direct and Overstock.

Top10_Mobile_pm_113015

The Mobile Retail Benchmark average slowed to 6.5 seconds, and the slowest mobile web site is now taking over 20 seconds to load.

Update: November 30th, 2015 at 3:30pm EST

Cyber Monday traffic continues its onslaught.  Confirming what we were seeing at noon, some retailers are seeing more traffic on Cyber Monday than on Black Friday.  Here is a view of real user visits coming from Dynatrace UEM from an online retailer.

UEM_vists_113015

I had made a mistake earlier when looking at this activity trend.  I had mentioned that typically we see Cyber Monday activity trail off after Black Friday.  This had been the trend for a couple of years, however when I look back at last years number we saw the same sort of trend.

600x285xindex2pmcybermonday-600x285.png.pagespeed.ic.t13VE8vkX_

We will continue to watch the days activity.

Update: November 30th, 2015 at 12:00pm EST

Cyber Monday traffic surprising as high as Black Friday for some retailers.  Here is some real world user data from a retailer showing that traffic is just as high as it was on Black Friday.  Interesting as usually Cyber Monday traffic trails Black Friday Traffic.

UEM_113015

Update: November 30th, 2015 at 10:45am EST

Having a look at Outage Analyzer, http://c8.net.ua is current having an issue which is impacting 220 different domains.  Not sure who or what this service is as it resolves back to an address in the Ukraine.   Retailers should have a look if their pages are calling this service.

Outage_Analyzer_113015

Keynote Internet Health is looking good for Cyber Monday.

Internet_Health_113015

Update: November 30th, 2015 at 8:40am EST

It has been a busy week so far.  Here is our first batch of numbers looking at the past 5 days for Cyber Monday.  We will start by looking at the Retail Home Page Benchmarks.  The companies on this chart are providing the Top 10 fastest retail home pages.  Apple, Amway and Staples still taking the top 3 spots.

Top10_HP_am_113015

The average across the Retail Home Page Benchmark improved to 4.7 seconds, and the slowest site took over 8.9 seconds to open.

In this chart we are looking at the top 10 retailers for online transactions, where we go to the home page, search for a product, look at the details, add it to a shopping cart and check out.  Apple, Amway and Fanatics take the top three spots.

Top10_TXN_am_113015

The average across the Product Order Transaction Benchmark improved to 29.4 seconds, and the slowest retailer was over 57 seconds

Finally in this chart we are looking at the mobile performance for online retailers.  These tests are executed from real wireless carriers from across the US.  The fastest mobile web retailers are Costco, Apple and Amway.

Top10_Mobile_am_113015

The Mobile Retail Benchmark average improved to 6.0 seconds, and the slowest mobile web site is now taking over 19 seconds.

Update:  Top three things Retailers can do when things go bad.

This Black Friday we’ve seen a couple of retailers have site issues.  At Dynatrace we’ve worked with hundreds of top retailers, and have thousands of customers using our services.  We’ve gathered a tremendous amount of expertise and best practices working with these retailers.  based on that experience here are the top three things retailers can do when things go bad.

1/. Have a plan.  Everyone needs a playbook.  Retailers need to make sure they have plan in place when things go bad.  The plan should not just be a reactive disaster recovery plan, that alone would be a bad plan.  The Retailer’s plan has to look beyond disaster recovery and make sure that their sites and applications have been thoroughly tested well before any holiday event.  The plans have to include ALL parts of the organization, it should not just be limited to operations, it has to include digital business owners, development, test and operations.

ruxit_dashboard
Dynatrace Ruxit Dashboard

2/. Have the right tools.  “Everyone has a plan until you get punched in the nose”.  Having a site go down during a holiday event is a traumatic experience for a retailer.  It’s not enough to have a plan, but you also have to have the right tools in place.  When things go bad for a retailer we typically see them go into a “war room” scenario.  Every team has their own set of tools which typically tell them that what they are responsible for is green, but still the problem persists.  Retailers need tools that bring teams together, they need a common tool that everyone in the organization can use including operations, development, test and digital business owners.

ruxit_smartscape
Dynatrace Ruxit Smartscape showing dependency relationships

3/. Understand the data.  Having the right tools in place is a step in the right direction, having the expertise to understand what the data is telling you is another thing.  Most retailers have teams of individuals who all understand their part of the application, and have a few “rock stars” who have the big picture in how it all works together.  Modern retail applications are incredibly complex, with applications having hundreds if not thousands of dependencies. Sifting through log files (which is what we see most often) is the old (and most costly) way of solving a problem. Retailers need to have an understanding of the relationship between application components and external internet factors.  While most retailers have a few rock stars what they need is a next generation set of tools to automatically notify that event is occurring, analyze all the dependencies and discover the root cause of the issue.

ruxit_rootcause
Dynatrace Ruxit Root Cause automatically analyzing millions of application dependencies.

Retailers can rely on these three things to see them through when things go bad.  The better they are at each of these the less likely an event will occur (an ounce of prevention is worth a pound of cure), the fewer war rooms they will have to convene, and faster they will be able to recover from an issue.

Update: November 29th, 2015 at 7:30pm EST

Still some activity over the weekend as retailers continue to offer promotions ahead of Cyber Monday.  Often geographic impact on response time gets overlooked.  Below we are showing the Retail Home Page Benchmark but aggregated by city.  New York sees the fastest response time followed by Atlanta and Chicago.  Kansas City, Washington DC and Mesa see the slowest response times.

Retail_City_112915

Since we are testing the same pages from different locations, regional content (local ads if applicable), Internet peering and CDN (Content Delivery Network) coverage tend to impact this geographic disparity.

Another way to look at geographic performance disparity is with real user monitoring tools like Dynatrace Ruxit.  Poor regional performance can be impacted due to traffic coming from oversubscribed CDN points of presence or even misrouted traffic.

Ruxit_UEM

Retailers can try Dynatrace Ruxit for free for 14 days from https://ruxit.com/

Update: November 28th, 2015 at 1:30pm EST

Here is a look at the US Retail Benchmark results for the past 48 hours.

Retail_BMK_112815

Here we can see 13 month trend for a Retail Home Page Benchmark starting from Oct 2014.

Retail_Trend_112815

Retailers in general are providing a slower response time since last year.

Update: November 27th, 2015 at 6:25pm EST

What does Black Friday look like for the Retailers?  Here is a last look at some real customer traffic being seen by Dynatrace UEM.  This is what retailers are seeing on there end; visits, landing pages, exit pages, conversions, bounces, user actions, client errors, etc…  Everything they need to make sure shoppers are having great digital experiences.

UEM_Traffic

We are going to take a break for a bit, but check back tonight and over the weekend for updates.

Update: November 27th, 2015 at 6:10pm EST

Having a look at the European results for Black Friday.   Dynatrace has a variety of retail benchmarks tracking from countries in Europe here are the top three by country in terms of web performance.

Country Retailer Response Time
United Kingdom
Tesco UK 1.1
LowPriceSHopper 1.2
Arcadia Group Limited 1.3
France
Shopzilla Aisle 0.9
IKEA 1.0
Leroy Merlin 1.3
Germany
Brands 4 Friends 1.6
Apple DE 1.8
Tichibo 2.2
Spain
Agapea 1.1
Zara 1.3
Ofertix 1.8
Italy
Oliviero 3.4
Bow.it 3.6
Media Shopping 3.8
Netherlands
HM 2.4
Lego 3.0
Brandos 3.7

Update: November 27th, 2015 at 5:45pm EST

Here are our third batch of numbers looking at the past 24 hours.  Again we will start by looking at the Retail Home Page Benchmarks.  The companies on this chart are providing the Top 10 fastest retail home pages.  Apple, Amway and Staples still taking the top 3 spots.

Top10_HP_pm_112715

The average across the Retail Home Page Benchmark slowed to 5.3 seconds, and the slowest site took over 13.5 seconds to open.

In this chart we are looking at the top 10 retailers for online transactions, where we go to the home page, search for a product, look at the details, add it to a shopping cart and check out.  Apple, Fanatics and Amway take the top three spots.

Top10_TXN_PM_112715

The average across the Product Order Transaction Benchmark slowed to 31.6 seconds, and the slowest retailer was over 75 seconds

Finally in this chart we are looking at the mobile performance for online retailers.  These tests are executed from real wireless carriers from across the US.  The fastest mobile web retailers are still Costco, Tiger Direct and Target.

Top10_Mobile_pm_112715

The Mobile Retail Benchmark average slowed to 6.3 seconds, and the slowest mobile web site is now taking over 20 seconds.

Update: November 27th, 2015 at 4:00pm EST

Black Friday continues to roll along.  Before our next update let’s have a look at why the top performing sites are doing so well.  When trying to uncover the “why” we look at KDIs (Key Delivery Indicators) like byte count (how heavy the page, or how much content is being delivered), object and connection count (how complex the page is) and Host count (the number of domains or third parties be called).  Have a look at the top performers compared to the bottom performers, can you see a difference?

Comparison_PM_112715

In every case, sites at the bottom of the list tend to have slower response times, because the pages are heavier (more byte count), more complex (higher object and connection count) and call more third parties (more host/domain calls).

The key to Black Friday success is delivery simplification and optimization.

Update: November 27th, 2015 at 2:00pm EST

Here is an update of the special retail categories we are tracking.  These are global results with tests being executed from various locations across the globe.  Consumer Electronic retailers provide the fastest results and Arts and Crafts sites in general proving the slowest response times.

CategoryView

Update: November 27th, 2015 at 1:00pm EST

Having a look at global retail benchmark activity. Brazil and China having the slowest online retail activity.

global_benchmarks

We are also monitoring some US based slow downs and issues, will report when we know more.

Update: November 27th, 2015 at 12:00pm EST

Here are our second batch of numbers now looking at the past 24 hours.  Let’s start by looking at the Retail Home Page Benchmarks.  The companies on this chart are providing the Top 10 fastest retail home pages.  Apple, Amway and Staples taking the top 3 spots.

Top10_HP_noon_112715

The average is still 5.1 seconds, and the slowest site took over 13 seconds to download.

In this chart we are looking at the top 10 retailers for online transactions, where we go to the home page, search for a product, look at the details, add it to a shopping cart and check out.  Apple, PCConnection and Amway take the top three spots.

Top10_TXN_noon_112715

The average product order transaction now takes 30.8 seconds, and the slowest was over 70 seconds

Finally in this chart we are looking at the mobile performance for online retailers.  These tests are executed from real wireless carriers from across the US.  The fastest mobile web retailers are Costco, Tiger Direct and Target.

Top10_Mobile_noon_112715

The average mobile retail site loads in 6.1 seconds, and the slowest takes over 19 seconds.

Update: November 27th, 2015 at 10:40am EST

Having a look at the Keynote Internet Health Dashboard.  This show peering relationship for major backbone providers.  I love this tool and so glad it is now part of the Dynatrace family.  Currently there are a few peering slow downs but nothing major for Black Friday shoppers in the US.

Internet_Health_112715

You can track this health report from http://internethealthreport.com

We are also tracking from Outage Analyzer and are seeing a couple of potential 3rd party issues, but nothing that would prevent shoppers from completing their retail transactions.

Outage_Analyzer_AM_112715

You can track this activity from http://www.outageanalyzer.com

Update: November 27th, 2015 at 10:00am EST

Things continue to do well for Black Friday retailers in the US, however the same cannot be said for other parts of the world.  UK Retailers are taking it on the teeth so to speak. (See what I did there?)  Black Friday started as a US based phenomenon, however as the digital world shrinks we are seeing other regions start to adopt this once only US tradition.

UK_HP_BMK_AM_112715

US Retailers have a certain amount of digital maturity when it comes to these events, and we expect to see this make its way into the rest of the world over time.  Global retailers have to pay attention to user experience as well.   For example here is a real time view from a retailer using Dynatrace UEM.  That’s a lot of yellow and red with a 4.45% conversion rate.  How much more revenue could a retailer generate if they switch some of those countries to green and increased their conversion rate by even 1%?

UEM_WorldMap

Update: November 27th, 2015 at 9:00am EST

Here are our first batch of numbers from the overnight results.  Let’s start by looking at the Retail Home Page Benchmarks.  The companies on this chart are providing the Top 10 fastest retail home pages.  Apple, REI and Staples taking the top 3 spots.

Top10_HP_AM_112715

The average was 5.1 seconds, and the slowest site took over 10 seconds to download.

In this chart we are looking at the top 10 retailers for online transactions, where we go to the home page, search for a product, look at the details, add it to a shopping cart and check out.  Apple, Fanatics and Costco take the top three spots.

Top10_TXN_AM_112715

The average product order transaction takes 31.1 seconds, and the slowest was almost 60 seconds

Finally in this chart we are looking at the mobile performance for online retailers.  These tests are executed from real wireless carriers from across the US.  The fastest mobile web retailers are Costco, Target and Tiger Direct

Top10_Mobile_AM_112715

The average mobile retail site loads in 5.9 seconds, and the slowest takes over 15 seconds

Update: November 27th, 2015 at 8:45am EST

Been busy this morning going over the numbers. Compared to last year retailers have turned it up a notch when it comes to availability.  We aren’t seeing any major site outages.  Good job retailers.  We are about to release some numbers in a few minutes.

Update: November 26th, 2015 at 11:50pm EST

Here is a real time dashboard looking at 10 of the fastest retailers based on transactional tests from the Dynatrace product order benchmark.  These are multi-step tests where our browser agents go to a home page, search for a product, add it to a shopping cart and go to checkout.

TXN_Dashboard112615

Apple is providing the fastest most consistent retail transaction.  Here is a more detailed view of how we are monitoring their applications from various Dynatrace Benchmarks.  We are executing home pages tests, transactional tests from multiple browsers and even mobile web tests.

Apple_Dashboard_112615

As an example of how we monitor retail transactions have a look at Apple’s retail transaction from one of the Dynatrace Product Order Benchmarks.

Apple_TXN_112615

Update: November 26th, 2015 at 8:00pm EST

Everyone will be wrapping up Thanksgiving dinner soon and as people start settling in to their couch surfing let’s have a look at some of the retail categories we are watching.  These results are for the past 48 hours.

Accessories and Shoes, average response time 7.5 seconds

Accessories_Shoes_112615

Apparel, average response time 6.4 seconds

Apparel_112615

Beauty and Skincare, average response time 5.6 seconds

Beauty_Skincare_112615

Consumer Electronics, average response time 5.9 seconds

Consumer_Electronics_112615

Department Stores, average response time 7.3 seconds

Department_Stores_112615

Hardware Stores, average response time 5.3 seconds

Hardware_Stores_112615

Home Furniture, average response time 5.8 seconds

Home_Furniture_112615

Home and Gifts, average response time 5.5 seconds

Home_Gift_112615

Sporting Goods, average response time 7.9 seconds

Sporting_Goods_112615

Warehouse Clubs, average response time 4.8 seconds

Warehouse_Clubs_112615

Watches and Jewelry, average response time 5.4 seconds

Watches_Jewelry_112615

Update: November 26th, 2015 at 2:30pm EST

As we head into the busiest online hours for Black Friday we will be showing lots of results from our proactive synthetic monitoring.  While these tests are coming from real web browsers let’s take a minute to talk about what is happening with the real users (consumers) who are busy shopping on the retail sites we are monitoring.

Real User data can highlight issues which wouldn’t be seen from just monitoring the server or monitoring with synthetic tests.  Dynatrace provides Real User data from tools like UEM (User Experience Management), DCRUM (Data Center Real User Monitoring) and Dynatrace NG/Ruxit.

uem_widgets

Above you can see how real user based data provides at a glance (and at a touch) visibility into geographic based performance issues (what happens when the retailers CDN is misrouting traffic), user satisfaction (what is causing customers to get a frustrated experience) and omni-channel visibility (how users are experience the site from a web, mobile web or mobile app perspective).

Below is a screen showing the last 7days of traffic from an actual online retailer.  This is all real user traffic.  You can see peaks and dips in the traffic as more customer are using the site.  This is a typical business day pattern for most online retailers.   If you look at the red highlighted areas we can see that an increase in client errors is occurring at the same time when we a decrease in conversion and a decrease in visit duration.

Retailer_UEM_example

Using these metrics this retailer can see in real time when their customers are being impacted by something and how that is impacting their conversion.

The view looks at a number of key metrics like User Experience Index, User Experience Breakdown, User Action Response Time vs Visit Count, Visit Duration, Conversion Rate, Client Error vs Bounce Rate.

  • User Experience Index is unique metric which looks at the combination of site performance, client and server errors for each visit tracked. Performance alone is useful but doesn’t tell the whole story.
  • User Experience Breakdown takes the visits and categorizes them by the User Experience Index so that customer visits get placed into buckets of those who received a satisfactory, tolerable or frustrated experience.
  • User Action Response Time is how long it takes the user to perform an action. This could be loading a page, touching a button, spanning a screen, etc…
  • Visits track a complete customer’s session from landing action (entry point) to exit action.
  • Visit Duration is how long a customer stays on the site for that visit
  • Conversion Rate is based on successful conversion rules determined by the retailer.
  • Client Errors are the number of errors being seen in the browser or mobile app.
  • Bounce Rate are the number of visits where a customer came to the site but did not do anything on the site before navigating away.

There are other metrics which can be included, but what we expect in the future is that retailers will look for tools which don’t just provide metrics to manually correlate issues, we expect them to be using tools which automatically identify when issues are occurring and the root cause behind the issue.  Retailers will move away from correlation and move towards causation.

Dynatrace Ruxit (shown below with some sample data) is an example of a tool which focuses on automated event notification and causation (problem identification).

Ruxit_JS_error

Dynatrace’s next generation Ruxit will automatically tell a retailer that a browser/client side issue is occurring, how many customers are being impacted by it and what javascript is causing the issue.  This immediacy of event notification and problem identification will allow retailers to pivot and addresses issues in a much faster fashion.   That velocity is vital for online retailers as our Holiday Survey found that 75% of all smartphone / tablet users and 81% of Millennial smartphone/tablet users report that if a mobile site or mobile app is buggy, slow or prone to crashes they would abandon it and shop elsewhere.

Update: November 26th, 2015 at 10:30am EST

Yesterday, one of my updates focused a lot on the browser/client delivery side of how we test.  In this update we will cover a little more about what happens on the server side.  As I mentioned in that update, reducing server side response times is one of the ways you can clear connections which allows you process more requests between the server and the browser.

This has a twofold effect for retailers.  First (as we mentioned in the previous update), it helps make pages load faster.  Second, it increases capacity, so that you can process more requests with less infrastructure.   We know from previous studies that improving response time leads to high conversion rates (generating more revenue) and optimizing infrastructure leads to reducing costs.   One of the most recent examples of how to optimize on the server side is being done through tools like Docker.  Docker provides a low cost way to scale applications, however Docker is a perfect example of the underlying complexity on the server side.  Docker containers include all the software required to run an application (operating system, web server, app server, middleware, database, etc…).   This software is typically exposed as hosts, processes, services, applications.  Understanding the relationships between all these components is one of the hardest things to accomplish.  We recently talked with a couple of development managers working on consumer facing applications and a couple of things they said really stood out.

“I can’t keep track of the stuff that changes”

“I don’t have a lot of time because I don’t have a lot of guys”

The underlying complexity on the server side is only going to increase; and as we hear operations and development managers will need tools to manage this complexity.

Below is screen from Dynatrace Ruxit.  This screen is the Smartscape view, here you can see how the server side complexity is represented

ruxit_smartscape

Given the complexity as seen above monitoring the responsiveness of the server is a useful thing.  One of the ways Dynatrace does this is by tracking the Server Think Time.  One way to do this is use W3C Request time as this give us a measure for the dynamic HTML content (excluding object requests for things like style sheets or images).  Here we can see that for the past 48 hours the top retailers with the fastest server side response times.

top5_serverside

The bottom server side performers all had W3C Request times over 0.5 seconds.

Remember for every retailer we report on over the next few days there is a stack of technology on the server side which is incredibly complex.  The challenge is understanding and keeping track of the dependencies and doing so with as few resources as possible.  Any one of those applications, services, processes or hosts could be a potential performance bottleneck.

Docker_(container_engine)_logo

We expect to see more and more retailers using platforms like Docker next year.  Without tools like Dynatrace Ruxit, managing this complexity will prove a challenge for retailers.

Update: November 26th, 2015 at 8:15am EST

I received this email yesterday.  A retailer was proactively apologizing for a slow down on their site.  Every touch impacts end user experience, and retailers who are aware of customer experience issues can be much more proactive in preventing them and reacting to them.

email_apology

Update: November 26th, 2015 at 7:50am EST

Some people are tweeting about potential issues with Walmart very early this morning.
Our analysis shows that Walmart is definitely on top of the issue, and deserves high praise for getting its digital performance back on track quickly.
Walmart_resuts

Update: November 25th, 2015 at 2:00pm EST

So while we wait for the Black Friday and Cyber Monday activity to peak, let’s take some time to explain how we will be looking at the sites on our benchmarks.   Dynatrace Synthetics uses real browsers located in locations all across the US to monitor the websites on our benchmarks.  Some of the benchmarks load the home page while others walk through a transaction where the browser opens the home page, searches for a product, looks at the details for the product, adds the product to a shopping cart and goes to the checkout process.

Loading a page sounds simple but there is an immense amount of complexity in how a page loads in the browser (we will talk about what happens on the server side later).   When we look at the response time of a page we often talk about the “over the wire” response time.  That is how much time it takes to load the HTML and all the subsequent requested page components (like styles sheets, javascript, images, etc…)  We also look at browser performance API timings; these are commonly talked about in the industry as W3C (World Wide Web Consortium) Navigation Timings.   These timings are markers that the browser can use to announce when specific events have occurred.  While in certain cases W3C timings provide interesting information they also provide a challenge as different browsers can execute them in different ways.

Response times give us an idea as to how fast the page loads, but there are factors which can impact these timings.  Some of these factors we call KDI’s (Key Delivery Indicators).  KDI’s are different from KPI (Key Performance Indicators) in that they provide the detail underneath the performance data.  A KPI is most commonly seen as performance, availability, consistency, etc…  A KDI is most commonly seen as byte count, object count, connection count, host/domain count, first byte time, DNS lookup time, SSL time, connection time, etc…   Optimizing the KDIs is one of the ways you can optimize the delivery of a page and improve performance.

Let’s look at it this way.  Below is an image of a page loaded in a browser and we are looking at the browser developer view.   This view allows us to get some insight into how the page loads.  The first item I will draw your attention to is the Timeline Network Requests.  That is the stack blue vertical bars.  This is shows the number of connections the browser can open at the same time.  The number of parallel connections is a real client side limitation that often goes overlooked.  Most browsers can only open 6 to 8 parallel connections which means that if you are using a large number hosts/domains (third parties) and calling a large number of objects, the browser needs to clear existing connections before it can open new ones.   Think of this as a real bottleneck.

web_page_build

Now optimizing for this bottleneck can be done in a number of ways.  Reducing the complexity of your page by reducing the number of hosts/domains (third parties), reducing the number of objects being requested, reducing the number of connection and (this is what Dynatrace Application Monitoring and Ruxit help with) reducing your server side think time.  The faster you can clear a request on the server side the more requests can be delivered to the browser.

In addition to the Timeline Network Requests, you can also see above a Timeline for Layout & Rendering as well as for Javascript & Events.  While good design can help with optimizing these, we often see developers try gaming the browser to get a fast “onload” time (this is a W3C navigation event).  One of the best descriptions on how to analyze and change the critical render path was written by Google’s Ilya Grigorik (Analyze Critical Path).  His description on optimizing the critical render path is all about simplification by reducing the number of critical resource, the number of critical bytes and the critical path length.  This makes a lot of sense.

When we execute our tests, we look at how a retailer can reduce the complexity of their pages. Every time we execute a test we record a tremendous amount of data around how that page loads.  This data looks not only at the W3C navigation timings but also the various KDIs we capture around every object request made as the page loads.  Below is an example of the same page as above and how we visualize a page loading in what we call a “waterfall” chart.   It shows KDI information, parallel requests, maps objects back to specific connections and IP addresses

waterfall_apple

OK, now that was really geeky.  Here is the “so what”.  Performance matters!  Our 2015 Holiday Shopping Survey told us that 49% of Holidays Shoppers will abandon a site/app if it fails to load in 3 seconds.  That number shows that consumers tolerance for poor performance is decreasing every year.  This applies to both traditional desktop web apps as well as mobile web applications.  This year 50% of millennial’s will do more shopping on their smartphones and tablets than they will by making in store purchases.

Almost all retail sites are “locked and loaded” ready to go in the next couple of days.  As we saw last year, we fully expect that those sites which have taken an approach reducing complexity and providing a highly optimized delivery of their pages will delighting their customers.

Keep watching here for updates over the next couple of days.

Update: November 24th, 2015 at 3:00pm EST

Grocery Store Activity.

So before we start looking at the retail activity this week, we need to look at some of the activity in the Grocery Store category as these sites are the ones which have been seeing an increase in activity as consumers make sure they have everything they need for their Thanksgiving meals.  Below is a view as to how the entire Grocery Store category is fairing from a mobile browser perspective.

Grocery_Store_Benchmark_112415

Currently (looking at the results for the past 5 days) we see the following sites taking the top performance spots.  We looked at 12 different grocery stores from locations across the US.

Desktop browser grocery store results

  1. Publix
  2. Stop&Shop
  3. Safeway

Mobile grocery store results

  1. Publix
  2. Stop & Shop
  3. Trader Joe’s

Travel Activity.

This is one of the busiest travel times of the year and when we look at the Dynatrace Airline Benchmarks we see a performance hit across multiple Airlines on Friday and Monday.  This is not unexpected as Airlines will have an increase of traffic this time of year as Holiday travelers check in online, or check travel schedules, etc…

Airline_Benchmark_112415

The above results show the impact that browser types still have on end user experience.  We see on average Chrome being slightly faster than Firefox and Internet Explorer.

Update: November 23rd, 2015 at 1:00pm EST

We are several days away from the start of the Holiday season, and everything kicks off with Black Friday and Cyber Monday. You may recall that last year the Dynatrace team provided a live holiday shopping blog following the web performance action in real time. We will be doing the same this year, with some new coverage that we didn’t have last year.

For some insights on how Millennials view mobile and web performance in the context of customer experience check out the results of our annual customer survey.

300+ new mobile and web performance tests

In addition to our traditional Top 50 Internet Retailer coverage, our team will be looking at more specific retail categories:

  • Accessories and Shoes
  • Apparel
  • Apparel Off-Price
  • Arts and Crafts
  • Baby
  • Beauty and Skincare
  • Consumer Electronics
  • Department Stores
  • Grocery
  • Hardware
  • Home Furniture
  • Home and Gift
  • Office Supplies
  • Pet Supplies
  • Pharmacy
  • Sporting Goods
  • Toys
  • Warehouse Clubs
  • Watches and Jewelry

In total we have included over 300 new web performance tests for both web and mobile retail sites. This year we will be looking at both traditional metrics like response/load time and availability and Key Delivery Indicators, including: byte count (page weight or how much content is being delivered), the number of objects and connections (complexity of the pages), hosts (number of third party contributors) and W3C request timings (server think time also known as first byte time).

This data represents actionable metrics that can be used to optimize the delivery of pages and applications to improve performance and, ultimately, customer experience.

One trend we already see when we compare the above metrics (at an aggregate level) is that since last year, the pages being delivered by the Retail industry (on average) have become slower, heavier, more complex, and are using more third parties.

Jones_1

Understanding that this is certainly a negative trend, it’s not fair for Dynatrace to single out the Retail industry because this is a general trend we see across most industries.

Jones_2

While 2015 is shaping up to be a banner year for online retail transaction, when it comes to customer experience and web performance we will be watching the above data and looking at social media to gauge what end users are saying about their online retail experience. Given that social media is a useful tool for retailers, many Dynatrace customers will be turning to customer experience metrics like the User Experience Index to understand user satisfaction.