Has healthcare.gov claimed “Mission Accomplished” too soon?

Over the weekend, those in charge of Healthcare.gov announced “Mission Accomplished” that the site is now performing optimally, and will be able to handle 50,000 concurrent users.   The healthcare.gov team also stated that if there happens to be a problem, they have software in place to help get to the root cause.  Any affected user will get a number in line and receive an email when they can return to the site.

When we tried to access the site today with our Ajax Edition to see what changes have been made, we received the following page. The question we have is “is this Mission really Accomplished?”

Figure 1- Mission accomplished?

It is only about the end user experience

The Healthcare.gov team based their success on performance metrics that only offer a limited server-side diagnostic view of problems.  That means that they are using internal, data center metrics to determine that problems are behind them, that error rates are down to 1%, and that response times are under a second.  The problem is that these metrics are not necessarily indicative that the end user experience has improved. Most of the issues that lead to customer frustration haven’t yet been effectively addressed.  Application experts know, backend server-side performance does not equal end user satisfaction.

Fastcompany.com recently wrote an article about how it is ONLY about the end user when it comes to managing performance. .  If the performance of the end user is poor, it does not matter what is done to fix the errors.  People will still be frustrated with the site and will continue to complain.  One study suggests that only 10% of users that have a problem even report it.

Let’s look at the end user performance for healthcare.gov from across the country for the past 30 days, from November 1st to December 2nd

Figure 2 – November 1st


Figure 3 – December 2nd

As you can see, the performance improvements haven’t had a huge impact from the end user perspective.  Looking at the graph below over the course of the last 33 days the response times to the end user haven’t changed drastically.

Healthcare.gov officials can’t yet claim sub-second response times

 Healthcare.gov officials can’t yet claim sub-second response times. The 400 bug fixes and upgrades to hardware and software haven’t been enough to calm user frustration.

Based on our outside-in testing, simulating use traffic from real user computers across the country, Healthcare.gov is not yet ready for volume traffic. We have shown in several posts here, here and here what the problems are with HealthCare.gov and how to address them. These fixes could have a profound impact to the end user if addressed. The data shows that even though the backend server-side “mission” may have been accomplished, the war for great end-user experience is far from over.