Update: Wednesday, July 13th 2016 10:35PM ET
So another Prime Day has wrapped up. I’ll be getting my $10 five pound bag of gummy bears soon. By all reports Prime Day has been a success for Amazon; TechCrunch reports that Amazon says it was their “biggest day ever” and sales were up over 60%. From a online delivery standpoint Dynatrace’s data has showed that Amazon rocked it. Based on our experience monitoring other top online retailers Amazon made some major changes in the way that they delivered their pages to end users compared to last year. Amazon optimized almost every end user KDI (Key Delivery Indicator) since the previous Prime Day. The amount of content (page weight/byte count) was reduced, the pages were simplified (fewer object requests and fewer client connections) and fewer third parties (external domains) were used. All retailers should take note how Amazon optimized their site, the team at Amazon should be congratulated for a job well done.
Update: Tuesday, July 12th 2016 5:35PM ET
It is after 5pm EST and from our perspective Amazon has had a very good day for CX and UX performance. All of our tests, homepage, mobile and transactional have shown very consistent results.
We’ve been using a number of dashboard visualizations to track things today and nothing has stood out as a particular problem. DNS, Latency, Server Response Time, Interactive time have all been very consistent. There were no outstanding geographic issues or issues with backbone ISP/Carriers. No third party impacted end user experience. For the amount of traffic Amazon saw today the performance has been very good for the majority of end users taking advantage of the Prime Day promotions.
Amazon’s web operations like all modern retail applications are exceptionally complex. Their application stack is based on a micro services architecture and managing the complexity and dependencies in these environments requires new views into cloud and new stack technologies. The view below is an example (not Amazon) of a new stack deployment in a next generation digital performance management platform (Dynatrace).
We will continue to watch Amazon into the evening.
Update: Tuesday, July 12th 2016 8:05AM ET
Quick Prime Day update. This morning as we check it looks like all things are a go for Prime Day. No noticeable impact on response times. We will check in with updates throughout the day.
Last week Amazon announced that Amazon Prime Day will be July 12. The team here at Dynatrace loves watching and analyzing the retail site performance of companies (something we’ve been doing for over a decade) to see how big brands stack up. There is much anticipation and excitement around this one-day event, when millions take to Amazon mobile or desktop in search of great deals. It’s a brilliant approach to clearing out inventory and testing new technologies and design approaches well ahead of the big shopping season highlighted by Black Friday and Cyber Monday.
Retailers will often create their own events and this is what the industry calls Flash sales. These are typically short duration sales events. Flash events tend to drive high traffic to sites and can provide retailers with valuable insight into how effective their IT strategies are.
This is important as study after study has shown that web and mobile applications which perform poorly for end users receive fewer conversions (conversions are goals that retailers use to track how successful they are). Fewer conversions most often translate into less revenue being spent online and if poor site performance is part of the reason it can also lead to brand damaging situations as frustrated customers take to social media to tell others about their bad experience.
We looked at a variety of “over the wire” KDIs (Key Delivery Indicators) which give us insight into how better site design and delivery can improve KPI (Key Performance Indicators) which allows us to understand how to improve end user experience.
Benchmarking Performance Over Time
Dynatrace’s Retail Industry Benchmark examines data over long periods of time. For Amazon, we can compare this year’s Amazon Prime Day to 2015 in terms of digital performance. The following chart shows the step-by-step timing for a real shopping transaction over the past 12 months. This test was run from multiple locations across the US with real Chrome based browsers.
We use a convention techies call “DOM Complete” to determine when a step is completed. It is commonly used in the industry, and indicates that the page is not only interactive for the end user, but also mostly loaded. Around the beginning of November 2015, we see a definite performance improvement from a customer’s perspective in how Amazon has designed and is delivering their shopping workflow.
Part of this improvement is due to Amazon focusing on optimizing their site for delivery. In the graph below, we can see that Amazon has trimmed about 1MB of page weight from their home page over the past year. Reducing byte count (page weight) is an easy design choice that can improve performance – you simply reduce the amount of stuff that has to download before the shopper’s page is loaded.
Amazon has also cut the number of objects on their home page by one-third, from ~300 to ~200. This means fewer back-and-forth interactions between your computer or smart phone and Amazon.com, and ultimately a faster page load. This approach also has the benefit of reducing the workload and possibly increasing capacity on Amazon’s site which means they can support more activity during peak times – like during the Prime Day promotion.
Amazon reduced the number of connections being made as well, from ~60 to ~40 on their home page, as shown in the following graph. Every time an end user loads a page the browser or mobile device opens connection to transfer data. If you open too few you can get a bottleneck, open too many you can overtask your server-side capacity (Server’s can only support a finite number of connections). Connections can be a limiting factor for scalability and server-side capacity.
In addition to changes in byte, object and connection counts, Amazon also reduced the number of external sites (“domains”) being called. Retailers connect to many external sites – to serve up ads and product recommendations, validate shipping, process your payment – each requiring behind the scenes interaction with another site. This adds to the richness and convenience of the shopping experience, but having too many third party connections can impact performance. It’s a balancing act. In the chart below we can see that Amazon has shed some domains compared to last year.
We also looked at Amazon’s mobile web presence. Delivering a mobile web site poses some challenges as the variety of devices that end users have access to is immense. End users expectations for mobile device performance is high, and the team at Amazon understand this. This is why we’ve seen their mobile site get much more optimized leading into Prime Day.
Here we can see that in the past 45 days Amazon’s mobile web page has also gone on a diet, shedding page weight (byte count).
Amazon has also optimized the delivery of their mobile web pages. Below we see that the Amazon mobile web homepage has decreased the number of objects, connection and hosts.
If you compare Amazon’s desktop homepage to their mobile homepage you can see that they are taking advantage of Responsive Design techniques to tailor content to the device the customer is using. You can visually see the differences between a page being viewed on a mobile device…
and how it is viewed on a desktop browser…
But there is more going on than the “over the wire” KDIs and responsive design.
Cache or Die
There are other techniques that Amazon is employing to ensure the shopper’s experience is not impacted by slow performance. A common technique is the use of client side caching, where certain static page elements (images, styles sheets, scripts, etc.) are stored in the browser cache. If done correctly, this substantially reduces the number of back-and-forth interactions between the browser and the site. Pages will load faster for shoppers, and the retailer’s site can handle more traffic.
The following images illustrate the effect of utilizing the browser’s cache. The first represents a page load with a cleared browser cache. Take note of the timeline in the browser developer bar along the bottom. There is a lot of activity when this page is initially loaded.
Here is the same page being loaded with a fully-primed browser cache. There is less initial activity, but the page is slower.
We can see exactly how Amazon’s caching strategy is seen from the shopper’s perspective – cached page performance is much faster than the un-cached performance.
The effect is most obvious when we compare the experience of first-time (uncached) visitors, whose browsers must download the entire page, to that of returning visitors whose browser cache already contains many of the page elements. Another way we can look at the difference between first time visits compared to returning visits is to break the traffic out by geography. This analysis is also useful to gauge the effectiveness of a Content Delivery Network (CDN), which should ensure consistently good performance by distributing static content closer to your site visitors. Here we see very consistent performance from every geography we tested.
Looking at the same metrics we have tracked since last year (byte, object, and connection counts) with and without caching, we can see how an effective caching strategy can positively impact response time and consequently visitor experience.
Retailers Take Note
Retailers should take note of how Amazon is focused on improving performance. We know from empirical studies that this leads to improved conversion rates, as well a cost savings from being able to handle more load with existing resources.
We will continue to follow Amazon leading into and through Prime Day 2016, so stay tuned.