In our last two blogs, we discussed what we have learned from last year’s holiday season as well as things that we can do in the preparation phase for this year’s upcoming event. In this blog we show you those dashboards and data points you need throughout the holiday season to make it a success.
The top goal for eCommerce sites is to ensure high conversion rates as this converts into business. IT’s responsibility is to ensure that consumers can use the eCommerce site in an “enjoyable” way. But there is much more than measuring the UpTime or Response Time of your services. The dashboards shown are taken from other eCommerce sites used to monitor the health of their application, infrastructure as well as end user satisfaction and conversion rate.
#1: Infrastructure and Application Health
Applications ultimately run on an IT Infrastructure; whether these machines are “physical”, virtualized, or running in the cloud. Ensuring a healthy infrastructure is the key requirement for IT. But it is more important to know whether there is an immediate impact on the hosted applications, services and processes. Before upgrading any hardware, reconfiguring your routing tables, or bouncing your application, it is important to understand whether it actually impacts the application and the end user. Just because you run on 95% of CPU doesn’t mean it’s a problem – maybe your developers just built a perfect system that consumes all resources available in an optimum manner.
You need a dashboard that alerts on system monitoring issues but also take into account the applications, services, and processes running on them. Are these impacted by the resource shortage or not? That answer dictates your action if you know what is actually impacted.
#2: Application Performance
The second dashboard you need focuses on the application, its performance, and impact on the end user. It answers the following critical questions:
- How much traffic is currently on the page? Is it still climbing? Is it outside the norm?
- Do we have an unusual high failure rate, e.g: failed credit card transactions, abandoned carts?
- What is the overall response time and is it violating my baseline?
- Is the application impacted by unhealthy app or web servers, e.g: high GC
- Are the hosts (physical, virtual or in the cloud) running into CPU, Memory or I/O limits?
- Are end users currently impacted when accessing the app? Are they leaving because of bad user experience?
- What is the current conversion rate or are we making money? 🙂
#3: Regional Availability and User Experience
The first 2 dashboards in this blog analyzed performance from within our datacenter. The dashboards above and below now focus on performance perceived from the outside – meaning – from the real end user perspective. You need to know if your app is not reachable from a specific region or when conversion rate drops even though your servers are doing fine. These two dashboards answer the following important questions for you:
- Is my site reachable from my key regional markets?
- If I am not reachable: How long did the outage last and did it impact Users?
- How many users do we have per region and what was their User Experience?
- How does the traffic per region develop over time?
- How is our conversion rate over time and how many orders do we actually get in?
#4: Real User Experience on the Conversion Funnel
You need a dedicated dashboard for all important actions along your conversion funnel. That includes landing pages and actions such as search, product details, add to cart and checkout. The dashboard helps you to understand:
- How many users you have on each conversion funnel step?
- Do they encounter problems during a particular action and is that the reason for a drop?
- How fast is each step and does it have an impact on end user experience?
#5: Third Party Monitoring dashboard
Most eCommerce sites rely on 3rd party content which not only impacts the feature set of the site but also performance and with that end user experience. Third Party Monitoring requires a view from two different angles: The third parties that are directly included into your website or mobile app and the external services you call from your backend.
These are the questions your Third Party Monitoring dashboard has to answer:
- Are the resources delivered via CDN fast or do we have regional problems?
- Is the integrated social media (Facebook, LinkedIn, Xing…) slow?
- Are the backend services facing bad requests to the integrate third parties?
- Is the performance of the third party good?
#6: Desktop Web vs. Mobile Web vs. Mobile App dashboard
Your potential customers can use desktops, tablets or smart phones to access your site. They will either have fast WiFi or slow dial-up speed. All of this impacts user experience. In order to analyze performance and optimize your site for these types of browsers, devices and connection speed you need a dashboard that tells you:
- How many users are accessing my portal via Mobile App or Mobile Browser?
- What are the top browsers used? Do we need specific optimized pages for older browsers?
- Do we need to optimize for lower bandwidths, e.g: use better image compression?
- Is there a difference between the Key Performance Indicators (KPI) depending on the different types of devices, browsers, mobile native vs. mobile web?
When disaster strikes: Collaborate with R&D
It is likely that you have smaller hiccups throughout the holiday season. To avoid lengthy and painful war room situations it is important to level-up your monitoring system and provide data your engineering team needs to speed up error resolution. Here is a list of capabilities that will speed up triage and error resolution:
- Capture all actions of each visitor
- Provide method level visibility on the server side including context information such as method arguments and return values
- Provide the ability for memory heap dumps and access to all requested application performance metrics, e.g: connection pool, thread count, heap sizes, …
- Use tools to capture this data that developers already use and that also allow sharing data from different environments.
Having these types of dashboards make it easy to monitor the success of the holiday season and also easy to react on problems and prevent larger damage by executing the right actions. Make sure you do not waste your time with problems that are not real-e.g: an individual user complains or trying to find a problem related to a regional outage of an ISP. Focus on those problems that impact a large number of users and that you can fix. This will make sure you keep conversion rate high and business flowing.
For further reading check out our other recent blogs such as DevOps Survival Guide: 2013 Online Holiday Shopping Season and With Confidence into the Holiday Season: Verifying Readiness in Test / Pre-Production