Testing and Optimizing of what I call “traditional” page-based web applications is not too hard to do. Take CNN as an example. You have the home page www.cnn.com. From there you can click through the news sections such as U.S, World, Politics, … – each click loading a new page with a unique URL. Testing this site is rather easy. Use Selenium, WebDriver or even a HTTP Based Testing tool to model your test cases. Navigate to all these URLs and – if you are serious about performance and load time – use tools such as YSlow, PageSpeed or dynaTrace AJAX Edition (these tools obviously only work when testing is done through a real browser). These tools analyze page performance based on common Best Practices for every individually tested URL.

This is great for traditional web sites but doesn’t work anymore for modern Web 2.0 Applications that provide most of their functionality on a single page. An example here would be Google and their apps like Search, Docs or GMail. Only a very small time is actually spent in the initial page load. The rest is spent in JavaScript, XHR Calls and DOM Manipulations triggered by user actions on the same URL. The following illustration gives us an overview of what part of the overall User Interaction Time actually gets analyzed with the current Best Practice Approach and which parts are left outside:

The initial Page Load Time on Web 2.0 Applications only contributes a small percentage to the overall perceived performance by the end user
The initial Page Load Time on Web 2.0 Applications only contributes a small percentage to the overall perceived performance by the end user

Let me explain the differences between these traditional vs. modern web applications and let me give you some ideas on how to solve these new challenges.

Optimizing Individual Pages has become straight forward – everybody can do this

Let’s look at CNN again. For each URL it makes sense to verify the number of downloaded resources, the time until the onLoad Event, the number of JavaScript files and the execution time of the onLoad Event handlers. All these are things that contribute to the Page Load Time of a single URL. The following screenshot shows the timeline of the CNN Start Page:

Key Performance Indicators that can be optimized for individual pages by optimizing Network Roundtrips, JavaScript and Rendering
Key Performance Indicators that can be optimized for individual pages by optimizing Network Roundtrips, JavaScript and Rendering

Now we can follow the recommendations like reducing the number of images, JavaScript and CSS files.

Optimizing Network Resources and with that speeding up Page Load Time for single pages
Optimizing Network Resources and with that speeding up Page Load Time for single pages

Once you are through the recommendations your pages will most likely load faster. Job well done – at least for web sites that have a lot of static pages. But what about pages that leverage JavaScript, DOM Manipulations and make use of XHR? You will only speed up the initial page load time which might only by one percent of the time the user spends on your page.

Single Page Web 2.0 Applications – that’s the next challenge!

I am sure you use Google on a regular basis, whether it is Google Search, Docs or Gmail. Work with these Web Apps and pay attention to the URL. It hardly ever changes even though you interact with the application by clicking on different links. Not all of these actions actually cause a new page with a different URL to be loaded. Let’s look at a simple example. I open my local Google Search (in my case it is Google Austria) and get to see the search field in the middle of the page. Once I start typing a keyword two things happen:

  1. the search field moves to the top and Google shows me an Instant Results based on the currently entered fraction of the keyword
  2. a drop down box with keyword suggestions pops up

But – the URL stays the same – I am still on www.google.at. Check out the following screenshot:

All the time while I execute actions on the web site we stay on the same page
All the time while I execute actions on the web site we stay on the same page

When we now look at the dynaTrace timeline we see all these Actions and the browser activities corresponding to these actions:

Executing a Google Search includes several actions that are all executed on the same URL
Executing a Google Search includes several actions that are all executed on the same URL

The question that comes up is – well – is the page performance good? Does Google follow their own Best Practices?

Analyzing Page Load Time is not enough

If we want to analyze this scenario with PageSpeed in Firefox we run into the problem that PageSpeed as well as YSlow are focused on optimizing Page Load Time. These Tools “just” analyze loading of a URL. In our Google Scenario it is just the loading of the Google Home Page – which with no surprise – gets a really good score:

PageSpeed and YSlow only analyze activities when loading the initial page
PageSpeed and YSlow only analyze activities when loading the initial page

Why this doesn’t work? In this scenario we miss all the activities that happen on that page after the initial page was loaded.

Analyzing All Activities on a page delivers wrong results

On the other side we have tools like dynaTrace AJAX Edition and Google SpeedTracer that not only look at the page load time but on all activities while the user interacts with the page. This is already a step forward but can produce a wrong result. Let’s look at the Google example once again: Whenever I strike a key, an XHR request is sent to the Google Search Servers returning a JavaScript file that is used to retrieve suggestions and instant results. The longer my entered keyword, the more JavaScript files are downloaded. This is “as designed” for the action “Search something on Google” – but it violates some of the Web Performance Best Practice rules. That’s why when analyzing all activities on a single URL will result in wrong results. Check out the following screenshot – it tells me that we have too many JavaScript files on that page:

12 JavaScript files from the same domain can be problematic when analyzing a single page load but not when analyzing a Single Page Web 2.0 Application
12 JavaScript files from the same domain can be problematic when analyzing a single page load but not when analyzing a Single Page Web 2.0 Application

How to test and optimize Single Page Web 2.0 Applications?

Testing Web 2.0 Applications is not as challenging as it used to be a couple of years back. Modern Web Testing Tools – both commercial and open source – provide good support for Web 2.0 applications by actually driving a real browser instance executing different actions such as loading a URL, clicking on a link or typing keys. I see a lot of people using Selenium or WebDriver these days. These open source tools work well for most scenarios and also work across various browsers. But depending on the complexity of the web site it is possible that you run into limitations where we need to consider commercial tools that in general do a better job in simulating a real end user, e.g.: really simulating mouse moves and keystrokes, and not just doing this through JavaScript injection or low-level Browser APIs.

For my Google Search example I will use WebDriver. It works well across Firefox and Internet Explorer. It gives me access to all DOM elements on a page which is essential for me to verify if certain elements are on the page, whether they are visible or not (e.g.: verifying if a the suggestion drop down box becomes visible after entering a key) and what values certain controls have (e.g.: what are the suggested values in a Google Search).

Object Pattern for Action based Testing

The following is the test case that I implemented using WebDriver. It is really straight forward. I open the Google Home Page. I then need to make sure we are logged in because Instant Search only works when you are logged in. I then go back to the main Google Search Page and start entering a keyword. Instead of taking the search result of my entered keyword I pick a randomly suggested keyword:

JUnit Test Case that tests the Google Search Scenario using Page Object Pattern
JUnit Test Case that tests the Google Search Scenario using Page Object Pattern

The script is really straight forward and fairly easy to read. As you can see I implemented classes called GoogleHomePage, GoogleSuggestion or GoogleResultPage. These classes implement the actions that I want to execute in my test case. Let’s look at the suggestionsFor method implemented on the GoogleHomePage class returning a GoogleSuggestion object:

In order to test the suggestion box we simulate a user typing in keys, then wait for the result and return a new object that handles the actual suggestions
In order to test the suggestion box we simulate a user typing in keys, then wait for the result and return a new object that handles the actual suggestions

The code of this method is again not that complicated. What you notice though is that I added calls to a dynaTrace helper class. dynaTrace AJAX Edition allows me to set Markers that will show up in the Timeline View (for details on this see the blog post Advanced Timing and Argument Capturing). The addTimerName method is a method exposed by a premium version of dynaTrace which we will discuss in a little bit.

Timestamp based Performance Analysis of Web 2.0 Actions

When I execute my test and instruct WebDriver to launch the browser with the dynaTrace-specific environment variables, dynaTrace AJAX Edition will automatically capture all browser activities executed by my WebDriver script. Read more on these environment variables on our Forum Post Automation with dynaTrace AJAX Edition.

Let’s have a look at the recorded dynaTrace AJAX Session that we get from executing this script and let’s have a look at the Timeline that shows all these actions that got executed to get the suggestions as well as clicking on a random link:

The Markers in the Timeline also have a timestamp that allows us to measure performance of individual actions we executed
The Markers in the Timeline also have a timestamp that allows us to measure performance of individual actions we executed

The Timeline shows every marker that we inserted through the script. In addition to the two that are called Start/Stop Suggestion, I also had one before clicking a random suggestion and one I placed when the final search result was rendered to the page. This timestamp-based approach is a step forward in tracking performance of individual actions. From here we can manually drill into the timeframe between two markers and analyze the network roundtrips, JavaScript executions and Rendering activity. The problem that we still have though is that we can’t really apply Best Practices such as #of Roundtrips as this number would be very action specific. The goal here must be to see how many requests we have per action and then track this over time. And that’s what we want – but – we want it automated!

Action/Timer based analysis that allows automation

Remember the calls to dynaTrace.addTimerName? This allows me to tag browser activities with a special name – a timer name. In the premium extension of dynaTrace, activities are analyzed by these timer names allowing me to not only track execution time of an action; it allows me to track all sorts of metrics such as number of downloaded resources, execution time of JavaScript, number of XHR requests, …. The following screenshot shows the analysis of a single test run focusing on one of the actions that I named according to the action in my test script:

Key Performance Indicators by Timer Name (=Action in the Test Script)
Key Performance Indicators by Timer Name (=Action in the Test Script)

This allows us to see how many network requests, JavaScript executions, XHR Calls, etc … we have per action. Based on these numbers we can come up with our own “Best Practice” values for each action and verify that we meet these numbers for every build we test – avoiding regressions. The following screenshot shows which measures dynaTrace allows us to track over time:

Multiple Key Performance Indicators can be subscribed and tracked over time
Multiple Key Performance Indicators can be subscribed and tracked over time

Instead of looking at these metrics manually dynaTrace supports us with automatically detecting regressions on individual metrics per Timer Name (=Action). If I run this test multiple times dynaTrace will learn the “expected” values for a set of metrics. If metrics fall outside the expected value range I get an automated notification. The following screenshot shows how over time an expected value range will be calculated for us. If values fall out of this range we get a notification:

Automatically identify regressions on the number of network resources downloaded for a certain user action
Automatically identify regressions on the number of network resources downloaded for a certain user action

If you are interested in more check out my posts on dynaTrace in CI – The Big Picture and How to Integrate dynaTrace with Selenium

Conclusion: Best Practices only work on Page Load Time but not on Web 2.0 Action Based Applications

It is very important to speed up Page Load Time – don’t get me wrong. It is the initial perceived performance by a user who interacts with your site. But it is not all we need to focus on. Most of the time in modern web applications is spent in JavaScript, DOM Manipulations, XHR Calls and Rendering that happen after the initial page load. Automatic verification against Best Practices won’t work here anymore because we have to analyze individual user actions that do totally different things. The way this will work is to analyze the individual user actions, track performance metrics and automate regression detection based on these measured values.