Over the past few months I have been giving online and offline talks called Performance Clinics. During these clinics, I show you how to analyze performance and give you feedback on the data you’ve collected on your own application. For Dynatrace users, I run a special offer called “Share Your PurePath” where I provide feedback and can give you extend Dynatrace licenses.

Every month I am going to write a short blog on the top problems found within the program so you get to see common problems that are out there and how to fix them.

This month’s stories are from Russell, an Operator of an online casino, and Joe, a Database Admin of a large telecom provider. They are Dynatrace Free Trial users and took advantage of my free performance clinics. Together we figured out why their applications are slow and how to fix them. This blog is to share the technical details around bad web page design, non-optimized deployment and a catastrophic database access layer implementation.

#1: Bloated Web Site of Online Casino

The first example is from Russell, who is in charge of hosting an online casino Web site written in ASP.NET. He frequently had to deploy a new version that came in from a 3rd party engineering team which caused him to spend nights and weekends getting his infrastructure in a state to fix and workaround stability and availability problems introduced with the latest version.

As he is not a performance expert when it comes to web development, he asked me to take a look at the data he captured with Dynatrace Browser Diagnostics which he found by reading one of my previous blog articles.

The following was my response to the data he sent me. It has evidence and tips for the 3rd party engineering team, and to Russell to deploy a much faster version of the Web site that won’t cause him so much pain to operate:

  • 9.66MB Total Page Size: this is too big for a landing page where you want people to continue clicking on the different online gaming offers they provide
  • 282! Resources: Most of these were high resolution images. There are techniques to optimize images from both size but also number of images on a single page. A great article was just posted by Tammy Everts on Image Optimization Checklist for Everyone
  • 8.8s Page Load: This was measured on my high speed office line. On an average line of people playing these games this time probably doubles or triples. It will significantly improve when optimizing these images
  • 95% Resources from Single Domain: Most images were delivered from their main domain. I gave him the advice to at least use some additional Image Domains such as image1.yourdomain.com and image2.yourdomain.com. Why? Because right now the browsers of end users can only download between 4 and 8 images at a time (depending on the browser). The concept here is called Domain Sharding
  • 1.8s in bad CDN connection: CDNs (Content Delivery Networks) are key for a good global online strategy. But if you pick a CDN that has problem like theirs, where most of the time is spent establishing a physical connection and not in actually downloading the content, then you should consider a different CDN provider.
The current deployment of this Web site will cause a lot of end user problems. Both engineering (reduce size and # of items) and Operations (better usage of CDNs for images) can improve the situation.
The current deployment of this Web site will cause a lot of end user problems. Both engineering (reduce size and # of items) and Operations (better usage of CDNs for images) can improve the situation.

#2: Catastrophic Database Access leads to High GC and Frustrated Business Users

This one came in from Joe who is a Sybase Database Administrator. For the last 2 years Joe has battled with the development team who always blamed his database instances to be the root cause for bad performance of their main conference room reservation website.

To prove their point, developers put in their own diagnostics code to measure execution time of their data loading implementation. Their timings showed that on average 45s were spent in loading the data – that’s why they pointed fingers to Joe. Joe on the other side had his Sybase diagnostics tools which showed him that none of these SQL Statements on average took more than 1ms. So – what should he optimize?

The following shows the pseudo code version of what the developers measured with their results compared to the results from the database:

Custom measurement by development was clearly showing a completely wrong picture of what was seen in the Sybase DB Tools.
Custom measurement by development was clearly showing a completely wrong picture of what was seen in the Sybase DB Tools.

Joe stumbled across of my blogs on database not always to be blamed for bad application performance. He downloaded the Dynatrace Free Trial and installed it on their Tomcat Server. As he was a database expert, not an application expert, he reached out to me to get my insight on the PurePaths he captured in their production environment. What we found was eye opening – not only for him – but in the end more so for the development team:

  • 24889 SQL Calls: Turned out that their loadDataForRoom was actually making 24889 roundtrips to the Sybase database. It executed 12444 individual SQL statements. Each of them on a separate JDBC connection which caused the Sybase JDBC driver to call an additional “set clientname” in order to initialize this connection with the right security context
  • All Data in Hashtable: The reason why they loaded that much data was because the original developer of that code was not an expert with SQL. The implementation of the “Data Access Layer” loaded each individual object property with an individual SELECT into a Hashtable. This fully loaded Hashtable was then used as the data repository for the getters & setters of the Data Objects.
  • High Memory and GC: Loading that much data into memory once wouldn’t be that big of a problem. Unfortunately this was loaded for every single web request and was not cached anywhere. The result of this was very high memory allocation per request and high Garbage Collection activity because all this memory needed to be cleared for the next requests that loaded the same data again in their own hashtables.
The Transaction Flow in Dynatrace shows the problem immediately. Loading too much data into memory causing high processing and CPU time on the application server.
The Transaction Flow in Dynatrace shows the problem immediately. Loading too much data into memory causing high processing and CPU time on the application server.

The next view I always check in these scenarios is the Database Dashlet as it shows all these database queries including Bind Values. In this case there were no bind values used as the developers generate unique SQL statements without using prepared statements. It is also easy to see that the JDBC driver called the set clientname 12444 times – once for each time the app took a connection out of the pool:

The Database View in Dynatrace gives full access of every SQL query including Bind Value and Connection Pool Details. Show this to developers and they know that something is wrong in their code
The Database View in Dynatrace gives full access of every SQL query including Bind Value and Connection Pool Details. Show this to developers and they know that something is wrong in their code

To show developers where this data was loaded and used I simply use the Methods Hotspot view in Dynatrace. It is easy to see that their Entity objects were accessing this one Hashtable which got created and filled for each individual request:

Methods Hotspot and the Caller Breakdown makes it easy to see where this Hashtable is used. The engineers can now rethink how to better implement their Entity Objects.
Methods Hotspot and the Caller Breakdown makes it easy to see where this Hashtable is used. The engineers can now rethink how to better implement their Entity Objects.

We had some more findings – especially the impact of memory allocations and garbage collection and how that also skewed their own measurement. A follow up step is to do some more analysis following my Hands-On Tutorial to Identify Memory Leaks. The good news is that these findings allow Joe to have a fact-based discussion about how to improve performance of the application instead of pointing fingers. Right now nobody blames his database anymore J

Let me Help you with your Performance Problems!

Are you dealing with application performance issues? It doesn’t matter if you are a developer, a tester, an architect, responsible for running an application or a business owner, feel free to contact me if you want any help and be sure to check out the following links:

Let’s work together to get your apps faster and let’s give back to the larger performance community by sharing some of our findings with the world.