TOC:

Cyber Week Wrap Up:  You Can Always Do Better

It has been a busy week analyzing the data from Black Friday and Cyber Monday. As we wrapped on Monday we did a comparison of the performance of top 3 retailers and the bottom 3 retailers.  Below are the results for the 24 hours of Cyber Monday.    While earlier we had discussed the impact of key delivery indicators like byte count, object, connection and host count, we do know that the more optimized a page is the better it will perform.

final_comparison

We further analyzed how the fastest pages load within a browser (using Chrome’s developer tools).  Here we can see that the faster pages spend less time loading objects, less time executing scripts and less time rendering.

Fastest

When we compare against a slow retailer, we see that it takes 10X more time to load page assets, 10X more time is spent on executing scripts and 3X more time rendering content.

Slowest

What our analysis shows is that there is always room for improvement.  In a highly competitive market, during a major shopping event, retailers must optimize or risk losing customers to their competitors.

Cyber Monday 5pm Update: A Busy Afternoon

It’s been a busy afternoon.  Traffic has picked up substantially and performance for some retailers has been impacted.

scatter_pm

Here are the top performers.

desktop_PM

Again, the top three spots are still Neiman Marcus, Dell and Apple.  These results are based on looking at the Average Total Page Load Time.  This time is derived from the W3C Navigation Timing API (this is not “over the wire” response time).

mobile_PM

Top three mobile web spots are Apple, Dell and Amway.  These results are based on the “over the wire” response time from an iPhone profile, using measurements from real wireless connections across AT&T, Sprint and Verizon.

The past couple of days I’ve been reviewing different things that retailers need to get right so my last note will be on understanding the performance impact on customers.

We showed you earlier this week how auto discovery and auto baselines can be used to detect the root cause of issues.  Below is a view of Dynatrace SaaS showing a CPU issue associated with a web server process.

process

Identifying an issue is certainly part of the problem, however you also need to understand the extent of the problem.  How many customers were impacted by the issue.  Having access to real time analytics is an important aspect to this.

Below is a view from an online retailer from earlier today.  This is a User Analysis view.  From this view a retailer can drill down to find out how many visitors are receiving a “frustrated” experience.  They can determine if those customers who had a frustrated visit had any common factors like channel/browser type, App version, OS version, bandwidth or geography.

user_analysis

While we’ve been spending most of our time talking about online and mobile web retail applications we need to also remember that instore retail applications can (and often do) suffer from poor performance during peak shopping events.

While some instore retail applications can be based off traditional web based technologies, sometimes they use ERP (Enterprise Resource Planning like SAP) based tools, or Citrix based retail applications.  These approaches also require some form of monitoring during Black Friday and Cyber Monday.

Below is a view of Dynatrace DCRUM (Data Center Real User Monitoring).  This part of the Dynatrace offering is focused around monitoring and decoding traffic which traverses through applications within a data center.  The decoded traffic can then be analyzed to find out if specific users or “bricks and mortar” locations are having performance issues.

dcrum2

Monitoring both external facing and internal facing retail applications is equally important during Black Friday and Cyber Monday.

Cyber Monday Afternoon Update: Have We Reached Peak Cyber Monday?

Mid afternoon on Cyber Monday and we are seeing lots more red on our Retail Benchmark Dashboards.

dashboard_mid_pm

To put this into context, lets’ have a look at how the top Retailers are performing across the country?  The view below is showing us the average performance from the Retail Benchmark across Dynatrace’s Last Mile network.   This is a network of real end user machines that run our proactive synthetic tests from every state.

last_mile

Here is a view of how real user traffic looks like for a US based retailer, where we see that they have had over 780 thousand visits (and going up) in the past 24 hours.

UEM

These geographic views are very useful at highlighting region performance problems, for example if a Content Delivery Network has an over subscribed point of presence or has misrouted traffic.

There is a lot of yellow and red being seen for many retailers as their applications are getting impacted during the busiest time on Cyber Monday.

Cyber Monday Noon Update: It’s All About Conversion

For our noon update let’s focus on some conversion details for Cyber Monday.  The following data is live real time real user data from a major US based retailer.  Below what we are looking at is a view of the number of successful conversions over the past week as display by digital channel (desktop browser and mobile web browsers).  The time of day definite impacts what types of channels customers use most.

conversion_channel

When we look at these channels in more detail we can see what browser type is being used most often for successful conversions.  Over the week (for the most part) Safari Mobile was used most often to convert (make purchases) for this retailer, however today Cyber Monday we see that Chrome (desktop) has started to eclipse Safari Mobile.   If you think about the demographics and behavior, this makes sense as most people last week were off work for the holidays (using mobile devices).  Today on Cyber Monday everyone is back to their regular routines (anyone shopping while at work today?).

conversion

Looking at conversion data in real time allows Retailer to understand (and react) to how performance is impacting their online revenue.  Taking it one step further the screenshot (this is not based on the retailer above) shows how real user based conversion data can be split into measures to track real time revenue, revenue by consumer, revenue by product, revenue by channel, etc…

revenue_analysis

Cyber Monday Mid Morning Update: Doing it Right.

For this morning’s update, I added some comparisons between some of the top performers and bottom performers from the Dynatrace Benchmarks.  I did this to provide some context between sites which a truly optimized for performance and those which are not.

I want to call out Apple as an example of a site which has been highly optimized.  Below is an example of a “waterfall” chart showing an individual page load for Apple’s Mobile Web Landing Page.  On the top of the waterfall is detail in terms of where the test was run from, what wireless carrier was used, what was the signal strength, what was the network speed, etc…

What makes the delivery of this page so compelling from a performance point of view is its simplicity.   There are only a handful of connections being made to 3-4 domains all controlled by Apple (there are no external third party calls).  We see that many of the object requests a being loaded in parallel.

apple_mobile

How does this benefit Apple?  First off it provides their customers a very fast experience as there is not a lot of time spent opening connections and waiting for objects to be downloaded.  This fast experience for customers reduces user frustration and makes it more compelling for the customer to remain on the site.  Secondly on the server side, the reduced number of connections and object requests takes the strain off Apple’s servers.  This means they can service more customers with fewer server resources (this speaks directly to increasing capacity and cost reduction).

Now let’s have a look at another company on the benchmark who is one of the bottom performers.  I won’t identify this company, but this is an example of how their design decision is impacting their customers experience.

comparison

When we compare Apple’s delivered byte count we see every time a new visitor loads this page 974KB of data is transferred, the bottom performer delivers 2,989KB (~3MB) of data.  Apple’s mobile page requests 43 objects, the bottom performer makes 423 round trip requests. Apple’s mobile site make 6 connections, the bottom performers makes 248 connections.  Finally Apple uses 4 domains/hosts where the bottom performer uses 105 domains (third parties).

Here is a view of their waterfall, I can’t show it in its’ entirety on this page as it is too big to fit, but feel free to click on this thumbnail to see it.

waterfall_big

While this is an example of extremes, it shows what is possible for Retailers in a highly competitive market.

Cyber Monday Update: Activity Picks Up after Quiet Weekend.

Here is a view of one of the dashboards we are using to track some key retailers.

dashboard_am

It’s been a quiet weekend. Most of the retailers have recovered from previous issues. As we did on Black Friday we want to focus on the example set by the top performers.

desktop_AM

Top three spots are still Neiman Marcus, Dell and Apple.  These results are based on looking at the Average Total Page Load Time.  This time is derived from the W3C Navigation Timing API (this is not “over the wire” response time).  I pasted in the bottom performers (anonymous) to put some context around just how fast the top performers are.  There is a 4X-5X difference between the top performers and the bottom performers.

mobile_AM

Top three mobile web spots are Apple, Dell and Amway.  These results are based on the “over the wire” response time from an iPhone profile, using measurements from real wireless connections across AT&T, Sprint and Verizon.  I include the bottom performers for context.

Weekend Update: No Major Issues

Here’s a quick view of how the weekend looked for the retailers we are tracking.

weekend

Black Friday Afternoon Update: Some Struggled, Some Excelled.

It’s been a busy afternoon. Several retailers had some issues, but we want to focus on the example set by the top performers.

w3c_retail_pm

Top three spots are still Neiman Marcus, Dell and Apple.  These results are based on looking at the Average Total Page Load Time.  This time is derived from the W3C Navigation Timing API (this is not “over the wire” response time).

retail_mobile_pm

Top three mobile web spots are Apple, Dell and Amway.  These results are based on the “over the wire” response time from an iPhone profile, using measurements from real wireless connections across AT&T, Sprint and Verizon.

How is it these companies can crush it year over year? Discipline in the optimization of delivery is certainly part of it. Having access to data and who is contributing to performance is another part of the solution.

Below is a view from Dynatrace SaaS of how various contributors can impact your customers.

analysis

Being able to automatically identify the contributing resources and then automatically baseline their performance over time allows Retailers to know exactly when they are impacting customer behavior.

Below is an example of how a Retailer can categorize the resources into content coming from a CDN (Content Delivery Network), content coming from a 3rd Party (social media, ad network, tracker, etc…) or content coming their own application (1st Party).

load_resources

We will continue to track our Retail Benchmarks results into the evening, over the weekend and into Cyber Monday. Keep checking back here for updates.

Black Friday Noon Update: Black Friday Strikes Again.

Noon update on Black Friday, and as per usual a number of retailers are running into issues.  This to be expected.  We’ve described at length the applications these Retailer’s create to service customers are exceedingly complex.

ops_noon

One of the Retailer’s site is failing because of server side issues on the backend, another because of network issues.  The faster you can identify these issues the less impact it will have on customers.

The image below shows how a retailer used Dynatrace SaaS to automatically identify a connectivity issue.

service_detail

While identifying issues automatically is great for operations, Retailers must not forget to prevent as many of these issues from occurring before they even get into production.  A well thought out Digital Performance Management strategy based on a DevOPs methodology is a sure fire way to prevent these issues.

I’m thinking of the line from the movie The Martian, except for retailers it would be…

I’m gonna have to DevOPS the shit out of this

It’s never to late to start thinking about how DevOPs can make your businesses better.

Black Friday Mid-Morning Update: Black Friday has Become a Global Phenomenon

Black Friday is no longer just a US based sales event.  Retailers across the globe are using the hype which started in the US and promoting sales to their consumers.  Below is a list of some of the countries/regions we track during Black Friday and Cyber Monday.  For each of these regions we are highlighting the performance of the Top 3 Retailers.  The results can vary from country to country as we are measuring from local markets.  I’ve also included the KDIs (Key Delivery Indicators) to illustrate some of the best practices these Retailers use for delivery.

Australia

Australia_pm

Brazil

Brazil

United Kingdom

UK

Taiwan

Taiwan

Nordics

Netherlands

Mexico

Mexico

Korea

Korea

Japan

Japan

Italy

Italy

India

India

France

France

Spain

Spain

Germany

Germany

Benelux

Benelux

Black Friday Morning Update: What are You Watching?

Good Morning!  Black Friday has been well underway since midnight.  Here are our first morning update as to who is providing the best performance so far.

w3c_retail_am

Top three spots are Neiman Marcus, Dell and Apple.  These results are based on looking at the Average Total Page Load Time.  This time is derived from the W3C Navigation Timing API (this is not “over the wire” response time).

retail_mobile_am

Top three mobile web spots are Apple, Amway and Dell.  These results are based on the “over the wire” response time from an iPhone profile, using measurements from real wireless connections across AT&T, Sprint and Verizon.

As the day progresses, what are some of the things Retailers should be watching out for? Here are some best practices.

#1/. Keep an eye on Active Sessions (below).  Every time a customer comes to your site it starts a session. If you see Active sessions plummet, that’s an issue.  It means that there is something wrong.  A drop in session traffic could be due to DNS or Networks issues if customers can’t reach your site.

active_sessions

The other metric to watch is User Engagement.  This can be measured in actions and session duration.  Actions can be anything from loading a page, touching a product, adding something to cart, filling a form, etc…  If your action count dramatically decreases or your session duration dramatically changes, it is a sign that that something is impacting your end users.  This could be a client side issue with a JavaScript.

#2/ Pay attention to the JavaScripts being delivered to your customers.  JavaScript is code which gets executed in in your customer’s browsers and mobile web browsers.  The interesting thing about JavaScript is that it can be delivered by both your teams as well as by some of your third parties.  JavaScript updates can change functionality on a page, JavaScripts can conflict with each other.

Below is a view showing how Dynatrace SaaS can auto-detect JavaScript framework changes.

events

JavaScript errors can lead to increased bounce rates (where customer prematurely leave your site).  Below is an actual example from earlier this week showing how an increase in Javascript errors lead to an increased bounce rate.

javascript

#3/ Geographic performance can point to issues.   If traffic is missing from certain locations, or performing poorly only in certain cities, this can be an indication of a CDN (Content Delivery Network) having a misrouted or an oversubscribed point of presence.

geo_locations

4./ Pay attention to peak activity intervals, Black Friday and Cyber Monday have different rules for when customers are going to hit your site.  If you are doing “flash” sales, make sure everyone know when they are going to be executed.  Below is a view from Dynatrace SaaS showing peak activity intervals.

peak_times

5/.  Analyze user behavior in real time.  Most user analytics tools provide data well after the fact and are not operational tools.  I had conversations with CMO’s who have told me that they don’t get their analytics data until 24-36 hours after a major event like Black Friday.

user_behavior

The above User Behavior view from Dynatrace SaaS shows how real time user analytics can be operationalized.

Understanding how customers are entering the site (Entry Actions) allows you to react to the effectiveness of your marketing campaigns and SEO.  What happens if your campaign is not directing customers to right page?

Understanding where customers are leaving your site (Exit Actions) allows you to react to performance bottlenecks and page errors which are frustrating your customers.

Understanding your bounce rate lets you know when a change is impacting your customers in a negative way.

Finally, establish conversion rules for your most important customer interactions.  Knowing when the functions that directly lead to generating revenue are having issues (and what is causing them) mean that you can reduce the duration of revenue impacting performance events.

6/. Analyze end user performance in real time.  Performance data can tell you if customers using certain browsers or mobile devices are having a satisfactory experience. It can tell you how much traffic is coming to the site, how the resources (images, stylesheets, JavaScripts, media, etc…) are performing for your customers.  It can tell you if third parties (ad networks, social media, trackers) are contributing to a performance bottleneck.

performance_analysis

Real time performance analysis can also be used to show the actual impact performance has on bounce rates.  Below is a view from Dynatrace SaaS that shows by the time customers wait 4 seconds 20% have already bounced off the site.

bounce_rate

It’s going to be a busy day, I hope some of these best practices help Retailers head off some costly brand damaging issues.

Check back throughout the day as more data comes in.

Thursday Afternoon Update: Looks Like Things are Starting to Cook.

Here is another view of how we are going to track some of the Retail results over the next few days.  This dashboard is showing us end-to-end response times for the entry/landing page for the retailers we are tracking by region/geography.

ops_dashboard

While the above view shows end to end time, one of the metrics we track is how fast the retailer’s servers are behaving.  By looking at W3C Request time this give us an indication of how fast the servers can respond to a request.

Below we are looking at the Top 10 fastest retailers by W3C Request time.  In this example we are looking specifically at their product search functions (how fast can they return a search result).

request

When monitoring a retail website, you have to look at more than just the home page to understand if they providing visitors with fast performance.

To illustrate this best, we look at more than just a landing page, we monitor a complete shopping process where we go to a landing page, search for a product, look at the product details, add it to a cart and go to checkout.

Below is an example of a retail shopping cart process leading into the holiday week.  Here you can see how this retailer optimized their checkout flow on Nov 11th.   This is a good best practice as it provides them with a couple of weeks to fix any issues ahead of Black Friday.

step_prep

When you monitor a business process, every click, every touch from a browser or mobile device executes a transaction.  Below is a view of a Transaction Flow Map from a retailer taken a few hours ago (Thanksgiving Noon), at which point we are seeing about 1.2 million calls being made to various Java, DB2, SQL Server and external tiers which support the responses to their customers.

Each of the search functions we looked at above is supported by a transaction flow which looks similar to this.

flow_map

Dynatrace’s Application Monitoring captures each individual request or transaction and stores the details associated with the transaction, this is what we call a PurePath.  Below is a view of how a PurePath can highlight where problem areas with a specific tire, method or database calls might be causing a performance bottleneck.

purepath

Retailers using this type of Digital Performance Management approach to watch every request/every transaction will have a competitive advantage over those who are not doing this to monitor every transaction.    When you combine the visibility we show in the Transaction Flow Map and the PurePath and then baseline the performance and apply some machine learning based algorithms, you can understand exactly when issues are impacting end users and automatically identify what caused them.

Below is a real example from a retailer taken earlier today showing a problem with an increase in failures coming from web requests to their images.   Online shopping is by its nature a visual exercise, and if your images are not being delivered that is a problem.

problem

I’ve been watching Black Friday retailer for years, and capabilities like this really excite me as it shows how real time data can be used to uncover issues before they become a brand damaging event.

We are keeping watch as we get closer to Black Friday.

Thursday Morning: Off to an Early Start!

Happy Thanksgiving!  Hours to go to the official start of Black Friday and Dynatrace is already seeing things pick up across the Retailers we are following.  We talked about this last year, that the start of Retail activity bleeds from Black Friday into Thanksgiving Thursday making this time of year more of a “Cyber Week” for Retailers.

Below is an example of some Real User data coming from major US Retailer.  The peaks and valleys in the graph show when there is more traffic to the site during business hours and less traffic in the early hours of the morning.

traffic

We will come back to this view over the course of the next few days, but look at the Conversion Count by Channel Tile (bottom left).  Here you can see Desktop traffic in blue and Mobile traffic in purple.  You can see the shift in time showing how mobile conversions occur later in the day during the week.  This supports previous studies showing that time of day impacts which channels consumers are using.

We expect that mobile web usage will be higher this year, however we’ve noticed an interesting trend over the past 12 months.  The Dynatrace Benchmarks show that mobile web pages have been becoming more complex and slower (on average) over the last year.

mobile_byte_trend

Since retail promotions are bleeding into Thanksgiving Thursday let’s have a look at some of the top performing mobile web retailers.

mobile_am

Apple, Amway and Dell take the top three fastest spots.

What Measures To Use?

Looking at some early comparisons, the question always comes up, what metric do we use to measure performance?  In the past the Dynatrace team decided to use “over the wire” response time.  This is how long does it take to retrieve all of the data being requested when loading a page/transaction.  However, some argue that other metrics like Average Total Page Load Time (marked by loadEventEnd) is a better measure.  Others argue that DOM Interactive (when a user can interact with the page) or DOM Complete should be used.  Still others claim that when a page begins to become visible (First Paint time) should be used.

The challenge with all of the above metrics is that depending on what browser you use (Chrome, Firefox, IE, etc…) the results will vary.  I highlighted these challenges in detail in responding to a comment on our SuperBowl 2016 coverage earlier this year.

timings

Above you can see a comparison of these metrics from an IE Browser point of view.   Here we see the top performers across IE are Apple, Tiger Direct, Costco, Dell and REI.  What we know is that the indicators we track around complexity hold up when we compare to metrics around performance.  Pages optimized for delivery perform better than those which are not optimized.

During the next few days we will continue to use “over the wire” measures of response times as they tend to provide a more consistent result.  That being said we will also use some of the other timing measures and I will make sure to point out when I do.

Cyber Week coverage continues.

Wednesday Afternoon Update: You want a piece of that pie?

Thinking about how many pounds you’re going to gain this thanksgiving?

While you’re busy thinking about how many pounds you are going to gain this Thanksgiving, retailers should be rethinking how heavy their pages are. When I talk about how heavy a page is, I am talking about how much data is transferred to a customer’s browser/mobile device when they load a retailer’s page/mobile webpage.

I don’t often rant, but I will for a second here. From time to time I hear from technologists that page weight is no longer an issue. “It’s ok, this content is being cached so there will be no performance impact.” Let me quickly explain that cache is content which is stored locally (your browser cache) or close by (edge cache from a Content Delivery Network). I don’t dispute that an effective cache strategy can dramatically improve performance, but it can’t be used as a crutch for not putting the effort into optimizing a retail site for delivery to a customer.

Here is some real data from a luxury retail brand from earlier this week:

returning

You can see that in a 24-hour period only 17 percent of its traffic was from returning visitors. These are visits where some content could have been locally cached. That means that 83 percent of traffic was from first-time visitors, which means that there were network requests for every single object (image, stylesheet, javascript, etc.) on the page.

This is expensive. From the performance standpopint, it is expensive because loading content across the network takes time. You can’t get around the fact that the more data you transfer, the longer it takes. While network performance is improving as more and more people have access to home fiber and LTE wireless, a lot of consumers don’t have access to this.

It’s also expensive to mobile users on fixed data plans. During a recent talk in Brussels, Klaus Enzenhofer, Director of Technology Strategy at Dynatrace, highlighted that heavy mobile webpages cost consumers as well. Klaus pointed to What Does My Site Cost to illustrate how heavy mobile webpages cost consumers.

mobile_cost

Above it shows that Canadian, Japanese and Brazilian consumers get impacted the most by the cost of heavy pages. However, the United States is not far behind.

Heavy pages are not only a burden for consumers, but for retailers as well. I mentioned earlier that another form of cache can be used, something called “edge cache.” This is where content is pushed closer to end users to speed up performance. Companies that provide this service are called CDNs (Content Delivery Networks). CDNs do not provides their services for free and for some larger retailers their annual spend on CDN services can be well into the seven figure range. CDNs provide good value for their services, but retailers, take note: consumption based pricing of CDN services can be impacted by your page weight. What that means is, if a retailer reduces their delivered page weight by 10 percent that can translate into a potential 10 percent reduction in their spend with a CDN. This will of course vary from contract to contract and from CDN to CDN, but make no mistake, optimizing the “over the wire” page weight provides retailers with some huge efficiencies.

What have top retailers learned?

Let’s turn to the Dynatrace Retail Benchmarks to confirm some of this. Let’s look at the average page weight for the top 10 fastest retail sites and compare them to the bottom 10 retail sites.

Top 10 fastest retail sites:                 1,732,313 bytes (~1.7 MB)

Bottom 10 slowest retail sites:          2,896,668 bytes (~2.9 MB)

That’s a pretty big difference. We see some sites with as little as 650KB data being delivered and some with as much as 5900 KB. What’s so fascinating to me is why haven’t those retailers providing slower response times looked at how their competitors and peers are performing to identify where they can improve.  Let’s have the same look at the mobile retail sites.

Top 10 fastest retail mobile sites:        810,780 bytes (~0.8 MB)

Bottom 10 slowest mobile sites:           2,480,965 bytes (~2.5 MB)

Why the difference?  This is likely due to responsive design optimization.  Some retailers identify traffic from mobile devices and provide a more tailored experience for those devices.

Optimizing delivery benefits both the customer and the retailer. Dynatrace has a team of experts as part of our Managed Insights organization run by Ben Rushlo. These experts live and breathe this type of analysis. They work with some of the top retailers across the globe. I’m scratching at the surface of what their analysis can unlock for you and your organization.

Check back later today as I look at some other key issues retailers should be tracking for Black Friday/Cyber Monday.

Wednesday Morning Update: What’s at Stake?

Two days to go before Black Friday. What is at stake? Aaron Rudger, director of our product marketing team, did some research: Deloitte’s Annual Retail Holiday Forecast (Sept. 21, 2016) shows that close to $100 billion will be spent online, and more than $600 billion in-store sales will be digitally influenced.

retail_outlook

The challenge, as we highlighted on Monday, is that the retail applications supporting this activity (be it online or in store) are exceptionally complex and getting more complex every year.

For example, below is a live view from a luxury brand retailer of what their retail application looks like from the server side (behind their firewall). We can see multiple services and processes all talking with – and depending on – each other. In many cases these services share the same host (you can think of a host as a machine) and compete for the same (limited) resources (CPU, memory, disk, network, etc.). At any point, any of the dependencies could be at risk and cause an issue that could impact other compute resources that touch a customer.

smart

Retailers need to understand and embrace this complexity. There is no single “server” to point the finger at any more.

What touches the customer exists on the end of a complex delivery chain, in which everything needs to work perfectly every time, or the retailer risks a brand-damaging experience with the customer.

deliery_chain

Retailers that embrace Digital Performance Management understand that managing this complexity is the IT challenge that keeps them from being competitive and prevents them from innovating.

In the end, it is always about the end user, the consumer…the customer.

If we look at the overall trend of end user-focused performance metrics for retailers, we see that year-over-year response times are improving in certain key metrics. I looked at First Paint time (how fast a customer would see something on the screen), DOM Interactive time (at what point a customer could start interacting with a page) and DOM Complete time (when the page had – for the most part – been loaded in the browser). This improvement shows us that the retail industry generally understands how important end user experience is to both their customers and the bottom line.

W3C_trend

While these metrics have, on average, been improving for the industry, there still remains a very wide gap between the retail industry’s top performers and the laggards when it comes to performance. Competition for a slice of that $1 trillion pie is intense.

We will be diving into that competition over the next few days. Check back here later today for some more insights as our live Cyber Week coverage continues.

Tuesday Update: T-minus 3 days to Black Friday

Understanding the potential obstacles to a successful online shopping experience, when it matters most

Yesterday we highlighted that on average, retail site performance has improved since last year at this time. We also showcased some findings on how poor performance leads to reduced online conversion rates, ultimately impacting revenue.

Today we are going to look at some of the metrics and details retailers need to keep on top of for Black Friday and Cyber Monday.

Industry Insight

Before we get into the details on how to monitor a retail application, let’s look at an update to a favorite chart from previous years.  This view looks across a variety of industries at their performance (according to the Dynatrace benchmarking practice) and compares them to specific KDIs (Key Delivery indicators).  We highlighted some of these for the retail industry yesterday. Here they are in the context of other industries.

KDI_2016

As we saw from the year before, overall IT complexity is increasing.  More content is being delivered to end user requests as web and mobile sites are becoming heavier with more connections and object request rounds trips being made.  In addition we are still seeing an industry trend to deploy more and more third parties (hint. keep an eye out for how the top retail performers handle 3rd parties, it will surprise you).

Only the top traffic, banking, sports and retail industries saw an improvement in overall response times.

Key Metrics Retailers Should be Following to Reduce Risk

Combining metrics onto the same screen (what the industry calls a “single pane of glass”) can be an effective way of managing the end user performance of your retail application.  Below are some examples of possible dashboards, retailers may be interested in seeing. Each will look at different metrics based on what is meaningful for them as well as what they are able to access. The key thing here is to start measuring from an end user point of view.  Measuring from just an internal (server) point of view provides an incomplete picture of what is happening

For example, some of the data we will be analyzing this week will be based on Synthetic tests run from the Dynatrace network.  Synthetics is one part of a complete Digital Performance Management strategy.   I created the view below as an example of what a Synthetic dashboard could tell a digital business owner in a single screen.

Key_Metrics

Let’s go over what these “tiles” would tell a digital business owner.

The Response Time & Success Rate tile (top left) provides a performance trend view which shows aberrations and events which could be impacting end users.  It’s also useful to know when you have recovered from an event and the degree to which performance has been change by an event.

The Geographic Response Time tile (bottom left) shows response time by specific regions.  This is important especially if you are using a CDN (Content Delivery Network) as high regional response times can be associated with an oversubscribed PoP (CDN Point of Presence) or misrouted traffic.  CDN services are not cheap and this is a way to help manage your technology investments.

The Contribution by Domain tile (second in from the top) highlights the impact that third parties like social media, ad networks, analytics/tracking tools, etc… are having on end user performance.  This view helps you manage technology investment and risk associated with a third party touching your customer.

The Key Delivery Indicators tile (second in from the bottom) shows observed byte count (how much data was delivered).  This often gets overlooked by retailers but will show issue related to un-optimized content (what happens when the creative team release a 10MB juggling monkey image to the landing page) or malicious activity (what happens when a hacker re-routes your site to their page).  Metrics like Object, Connection and Host count also provide an indication as to the complexity of the site and if something unexpected is occurring.

Let’s switch to the right side of the screen.

The DNS performance tile (top second from the right) shows DNS resolution time.  DNS can be thought of as phone book, routing site names to server addresses.  Again this often goes overlooked, however the DDoS attack on DNS provider Dyn on Oct 21st shows that DNS is critically important.  Knowing when/if your DNS is being impacted allows you to make changes and recover faster.  It also allows you to understand if you are investing in the right partner for providing DNS.

The Network Latency tile (top right) is a measure of how healthy your network connections are.  This data can be used to understand if you peering issues with your network providers or if your network infrastructure (load balancers) is under pressure.

The Server Response time tile (bottom, second from the right) is a measure of how fast the server can respond to a request.  This allows you to understand from an end point of view if the server applications are causing a performance bottleneck (and if you know anything about Dynatrace you know that is where Dynatrace SaaS, AppMon and DCRUM shine when it comes to diagnosing problems).

The last tile on the bottom right, shows Client Render time.  This allows you to understand how long does it take for the browser/mobile browser to display something for the end user.  Understanding what is happening in the user’s browser is the final link in the chain.

Using Real-Time Real User Data to Protect Online Revenue

While a Synthetic only view can provide a lot of visibility, it should not be counted on alone when managing an event like Black Friday and Cyber Monday.  Retailers also need to have access to real user data.

Real user data puts context to the synthetics data.  It allows you to see how a performance event impacts real user’s behavior.  It allows you to quantify behavior as it relates to bouncing off a site or converting a user to purchase an item.  Below is an example of a real user dashboard.  This dashboard is based on a “gap” free view of every user visit with every click, swipe, touch, etc… that a user does being captured.

Business_Analytics

Let’s go over what these “tiles” would tell a digital business owner.

The first tile in the top left is the World Map which tracks User Experience Index, APDEX and Traffic. The User Experience Index combines things like response time with error and other conditional measures to provide an overall gauge of the health of the end users visits.  APDEX is a page performance index used in the IT industry.  These measures tell when and where there are performance issues being seen by your end users.

The User Analysis tile (middle top) shows you how many customers are getting a satisfactory experience, as well how many are getting a tolerating or frustrated experience.  These categories can be filtered to see if frustration levels are being impacted more so in certain geographies, using different device types or how users connect to the internet.  This tile also provides visibility into bounce rate (users leaving the site prematurely) and conversion for the site.

The Omni Channel tile (top right) provides you with traffic trends from web/desktop users, mobile web users, mobile app users and bot traffic.   This allows you to see where traffic is coming from and if your users are having performance issues on certain channels of interaction (ie. are mobile users having worse performance than desktop users).

The Top Landing Pages and Top Exit Pages (bottom middle) show you how consumers enter and exit your sites.  This data can help validate SEO and campaign investments.  This data can also help you understand if performance bottlenecks or errors are causing your customers to leave your site.

The Real User data approach when used to its fullest potential can provide a digital business owner with visibility into how much revenue is being generated online in real time.  This helps quantify the lost opportunity cost for a performance event.  It can also be used notify the business when expected revenue goals are not being met by sending alerts and triggering incidents if revenue drops below a certain value.

Revenue

In my next update we will look at ways to automate the monitoring of these highly complex retail applications.

Cyber Week Kickoff, November 22 – 2016 Holiday Season Analysis

With the holiday shopping season kicking off later this week, the big question is: Will IT Complexity Jeopardize Retailers’ Digital Performance this Black Friday/Cyber Monday?

According to recent data, this Cyber Monday is expected to be the largest online shopping day in history, with sales estimated to hit $3.36 billion. There’s a lot at stake, including revenue and brand reputation

Dynatrace will be tracking the top 50 U.S.-based retailers from major markets across the country to see how this peak season impacts the speed and availability of their websites and mobile apps. Why does it matter? Because site speed impacts conversions. Nordstrom reports, a mere 0.5 second increase in response time reduces conversions by 10 percent. A shocking statistic to say the least!

1011_IG_BlackFriday_SiteSpeed_720x480_FINAL2

Our take: Retail websites are faster, but also heavier and more complex

This year, we’re most concerned about the increased complexity of retail applications. The interdependence of their digital ecosystems could cause them to  buckle under the pressure of peak holiday shopping.

We conducted a year-over-year analysis of the digital performance of these top 50 U.S. retailers to analyze this increasing complexity and how retailers have been tackling it.

The good news for shoppers (and retailers) is that their average year-over-year speed has gotten just a little bit faster – by almost .03 seconds – from 5.07 seconds in 2015 to 4.78 seconds in 2016.

1011_IG_BlackFriday_2015-2016_720x480_FINAL

The bad news is that, as retailers compete to deliver richer and more engaging experiences, their websites have become heavier, more complex and more interdependent on third parties. Our year-over-year analysis found:

1011_IG_BlackFriday_Weight_720x480_FINAL

Why does increased complexity matter, anyway?

Because heavier and complex websites are much harder to manage, and introduce greater operational risk. This means that, if/when things go wrong, it’s exponentially harder to pinpoint the source of the problem and then fix it. And, while companies are busy trying to mend the problem, frustrated consumers will up and shop elsewhere. Bottom-line: complexity presents a risk to retailers’ reputation and revenue especially during the holiday shopping season when digital performance matters most.

How are retailers addressing this?

Since so much of this application complexity lives on the server side, that’s where we’ve seen retailers working most to optimize performance. They’ve been increasingly adopting container technology to improve and streamline performance.

Per our July study, the five fastest U.S. retail sites all load in under two seconds and site complexity is kept to a minimum. The group’s average figure for hosts, objects and page size all sit below global standards. Clearly, for the top end of the U.S. retail industry, speed is a key digital performance indicator; over this holiday weekend, we’ll see how well all the others hold up.

Image insert

Our Live Blog Coverage for Black Friday & Cyber Monday kicks off today!  Bookmark this page and check back here for updates.  Our team won’t be taking Thursday and Friday off so we can provide you with the latest details on how Retailers web and mobile sites are performing this year.  Also feel free to follow my real-time updates on twitter @davidlewisjones.