Many Americans are looking forward to the new Healthcare website that allows them to select the health insurance best for their needs. As with any new website that has been anticipated by a lot of people, it was not a big surprise there were glitches when millions of citizens tried to use the new portal after its launch.

Now – there are many different reasons why websites that need to scale for that many users don’t deliver on the promise of good end user experience. A general “cultural” problem is that performance and scalability are pushed towards the end in favor of more functionality resulting in problems that don’t allow the end user to consume these great features. Changing this culture with the support of tools that integrate in your continuous delivery process is mandatory to avoid these types of problems. We have blogged about this in the past based on discussions we had with companies that made that transitions.

Let’s put the spotlight back on this key Obamacare website.  At the time of this posting, I’m not privy to data collected from within the site’s production datacenter environment. That kind of code-level visibility would help us find the root cause of problems immediately.  However, we can still learn quite a bit by using our freely available tools that analyze how web pages and third party objects are rendered within real end-user browsers.  Our goal is to quickly find the top problem patterns that help us figure out why so many people are complaining.

The Analysis

One of my colleagues walked through different use case scenarios on healthcare.gov and sent me his AJAX Edition session files for analysis. Here is an overview that shows that most pages lack basic WPO (Web Performance Optimization) aspects:

Bad WPO ranking on most pages and lacking behind the TOP US healthcare sites
Bad WPO ranking on most pages and lacking behind the TOP US healthcare sites

Now let’s have a closer look and highlight the key observations from these sessions.

Observation #1: Homepage impacted by Initial HTML and 3rd Party Content

Looking at the Timeline View for loading healthcare.gov shows some very interesting things. It takes a long time to download the seemingly small 59k initial HTML document. This is probably caused by bandwidth constraints on its web server, as most of this time is contributed to the server response time and not the network. What is even more interesting is the use of multiple different 3rd party monitoring solutions such as Google, Chartbeat and Pingdom. All of these 3rd party components need to load JavaScript files that impact the initial page load time.

Good news is the first visual impression is good with being under 2 seconds. But – there is definitely the option to optimize overall page load by optimizing the use of 3rd party components and making sure the initial HTML page can be served faster from the web servers.

Most impact by slow download of HTML page as well as lots of content from many 3rd party components
Most impact by slow download of HTML page as well as lots of content from many 3rd party components

Observation #2: NO CSS and JS Merging on Registration Page

The registration page – https://www.healthcare.gov/marketplace/global/en_US/registration – is actually a very bad example of some of the well-known practices of Web Performance Optimizations. It seems they forgot to merge CSS and JS files together as they are currently loading about 55! Individual JavaScript files and 11! Individual CSS files! The following screenshot shows the list of individual resources that could all be merged – especially because many of them actually belong logically together as well, e.g.: jQuery and jQuery Plugins.

Loading too many small JS and CSS files instead of merging them together. This results in too many roundtrips to the web server
Loading too many small JS and CSS files instead of merging them together. This results in too many roundtrips to the web server

Besides not merging files they also use some non-minified versions of the jQuery plugins and use rather large and uncompressed versions of their images.

Observation #3: Server-Side issues with AJAX calls on Profile Page

The most obvious end user performance impact was seen on the My Profile page. We could see a 16.8 second server response time for an AJAX call that returned some basic user information for the logged in user. The end user has to sit and wait until that AJAX request finally completes so that the page is fully operational. The timeline shows how much impact this AJAX request really has:

While waiting on the AJAX Response the web UI is “blocked” and user has to wait
While waiting on the AJAX Response the web UI is “blocked” and user has to wait

What’s even more interesting is that every interaction on the MyProfile page re-sends this AJAX Request returning basically the same information again without caching it. Taking a closer look at the actual content that is returned, it seems that about 95% of the content is always the same (e.g: name, phone number, …). There is only one field in that returned JSON object that actually changes. The question is whether this can be optimized to cache that static information (Name, Phone, …) and only return the dynamic values. This would also speed up server-side processing, reduces network package size and improves end user response time:

This AJAX call takes up to 16.8s on the profile page and is executed for every user interaction. Optimizing this logic will improve end user response time and takes pressure from the application servers
This AJAX call takes up to 16.8s on the profile page and is executed for every user interaction. Optimizing this logic will improve end user response time and takes pressure from the application servers

Observation #4: Heavy JavaScript Processing by Backbone, Underscore, … JS Libraries

Filling out an application uses a lot of JavaScript to deal with validating input as well as with presenting the results. The usage of 3rd party JavaScript libraries is very common as it takes a lot of work off UI developers. On the other side these frameworks might not always work perfectly for a particular application. Looking at the JavaScript hotspots on that particular page shows some very significant hotspots by the click event handlers for things like Add Deductions, Annual Income Information or simply clicking on the Accept Warning and Continue. Response times of up to 7 seconds were mainly caused by these JavaScript frameworks that iterate through the whole DOM to identify those elements that need to be modified or not. Depending on the browsers JavaScript engine this can have a significant performance impact as it was in the case of my colleague.

The more complex a page the more effort for JS Frameworks to dynamically iterate through the DOM. In this case we have several seconds of pure JavaScript Execution Time to modify the DOM
The more complex a page the more effort for JS Frameworks to dynamically iterate through the DOM. In this case we have several seconds of pure JavaScript Execution Time to modify the DOM

The ToDo List in order to “Get Well”

Looking at these observations there are several points for a ToDo List in order to make the site faster and more responsive:

  • Rethink the 3rd Party components. Delay loading of these components if possible to not impact initial page load time
  • Use minified JavaScript and CSS Files to save on bandwidth
  • Merge JS and CSS files to reduce the number of roundtrips
  • Compress Images to save on network bandwidth
  • Optimize the server-side performance for these long running AJAX Requests
  • Make the pages “slimmer” (less DOM elements) to speed up JavaScript executions that iterate through the DOM

In general, it is important that you optimize your website for performance based on the anticipated load from its inception, to the website design, and all the way through the development phase and roll-out.  This is not only important for application launches but anytime you have higher peak load events, which we call “Rush events”.  You not only need the solutions, but the framework and step by step best practices to plan for peak events or launches.  Compuware has developed best practices guides, customer stories of success and expert advice around optimization in performance in a free program called “The Compuware APM Rush Program.”  Visit this site for access to the expert advice and methodologies to ensure your launch or rush event goes flawlessly.

More Things to Consider …

Those of you that know our blog are familiar with all the problem patterns we have talked about in the past. This list of problem patterns just adds to the list – but in fact – there is not a whole lot of new concepts here. It seems that we keep seeing the same problem patterns over and over again. If you are responsible for end user experience, performance or scalability of your Web, Mobile, Desktop or any type of other application that needs to deliver value to the end user have a look at our top blog posts such as Top 8 Performance Landmines, Balancing the Load or The DevOps way to Solving JVM Memory Issues.