I keep getting questions about how to best analyze memory leaks – especially when they are not always reproducible by the developer on the local workstation. If you never experienced a memory leak issue (or you simply don’t admit it) then read up on some real life examples on our blog: Fixing Memory Leak in Java Production Environment or DevOps way to solving Memory Issues.

If you want to test this out on your Java or .NET Application follow my steps in this blog, watch my webinar on YouTube where I describe both Memory but also Thread Diagnostics with Dynatrace or read the technical documentation on Memory Diagnostics.

Step 1: Tooling – We use the Dynatrace AppMon Free Trial

If you don’t yet have Dynatrace register for your 30 Days Free Trial with the option to get a lifetime license for your local machine -> Learn more with our Share Your PurePath Program!

Step 2: Connecting your JVM or CLR

The best way is to simply follow the steps in the tool to setup a new Application Environment. There is a step-by-step guide on how to connect an agent into your Java or .NET Application on the Dynatrace Community.

Step 3a: Memory Analysis Server

It is recommended to install and use a dynaTrace Memory Analysis Server. It should be installed by default but may not run automatically. Therefore check your services on windows and make sure you launch the dtanalysisserver on Linux. Before you launch it, you want to edit dtanalysisserver.ini and change the –Xmx to the size of the heap you want to analyze, e.g: -Xmx4G to analyze a 4 GB Heap.

Step 3b: Pre-Requisites for .NET Memory Monitoring

Info: Since dynaTrace 6.1 the option “Automatic” means that the .NET agent tries to turn on the necessary profiler callbacks on the fly and then switches them off after the memory dump is finished so this way the application doesn’t have to be restarted. This temporary switching works only with .NET application from version 4.0 running in blocking Garbage Collector mode.

This step is just for .NET Environments. Due to the nature of .NET there are two things you need to ensure:

  1. The user that runs your .NET Process is allowed to capture windows performance counters. The user must be in the windows group “Performance Monitor Users”
  2. You need to enable .NET Memory Dumps in Dynatrace through the System Profile -> Agent Group -> Agent Mapping -> Advanced Dialog
.NET Only: Enable Memory Dump Collection for .NET Processes
.NET Only: Enable Memory Dump Collection for .NET Processes

Step 4: Trending Dumps to Detect Problematic Classes

Once you have a Java or .NET agent connected to dynaTrace you can start creating memory dumps. We need to open the Total Memory Dashlet which can be done either through the Cockpit (tree view on the left of the client), Start Center (Start Button in the toolbar – > then select Memory Diagnostics) or through the Process Health dashboards:

Open Memory Dump Dashlets
Open Memory Dump Dashlets

Tip: If your button is greyed out it means you don’t yet have an agent connected. Go back to the Application Environment Wizard and walk through connecting your agent

There are two types of dumps: Memory Consumption Trending and Deep Memory Leak Analysis. We start with the trending dumps as the goal here is to create several of these “Lightweight” dumps to figure out which objects from which classes are growing over time:

Capturing several Lightweight Memory Dumps to identify leaking classes. Change frequency. Uncheck Force GC to identify all objects on heap and not just “leftovers” after GC
Capturing several Lightweight Memory Dumps to identify leaking classes. Change frequency. Uncheck Force GC to identify all objects on heap and not just “leftovers” after GC

Tip: I typically uncheck the box “Force GC before creating snapshot”. Why? Because I want to see all objects on the heap which allows you to also identify objects with a high churn rate. With the option checked we would only see objects that are still referenced after a GC run.

Once you have the trending dumps you can either analyzed one of these dumps individually to inspect number of instances per class and also the amount of memory allocated.

View all classes and instances on the heap. Tabs on the bottom are only filled with data for Deep Memory Leak Analysis Dumps.
View all classes and instances on the heap. Tabs on the bottom are only filled with data for Deep Memory Leak Analysis Dumps.

But even better is if you select multiple dumps and then watch the trending line to see whether memory is increasing over time:

The trend line of the selected dumps shows whether you have a potential memory leak
The trend line of the selected dumps shows whether you have a potential memory leak

Tip: If I see a trend like this I select the dump with the lowest and highest usage and click on compare. You can select multiple and compare but we get similar results by comparing the two dumps with the biggest difference

The Delta columns show which classes grew in size and count. Focus on those that are significant for your app: custom classes or things like Hibernate, SharePoint …
The Delta columns show which classes grew in size and count. Focus on those that are significant for your app: custom classes or things like Hibernate, SharePoint …

Tip: It doesn’t make sense to focus on String or other basic types. It’s better to look at custom classes or framework classes such as Hibernate, SharePoint, etc., as these classes ultimately reference the primitive types. Start searching in the dynaTrace Client by clicking in the table and start typing e.g: com.mycompany. All views have this built-in filtering capability.

Step 5: Detecting Memory Leak Root Cause

Once you have identified classes through the trending dumps that you believe shouldn’t grew that much in size and count it is time to do a deep dive memory leak analysis. For this we create another dump – this time selecting the Deep Memory Leak Analysis option – optionally with capturing Strings (Java Only) and Primitive values:

Capturing a full memory dump including values of primitive fields and string objects (Java Only)
Capturing a full memory dump including values of primitive fields and string objects (Java Only)

Tip: Capturing such a memory dump may take up to several minutes as lots of data is getting captured. In production environments this has to be used with care.

Now it’s time to analyze the captured instances – focusing on those classes we identified through the trending dumps. The following screenshots show you how to walk the referrer trees in order to figure out which objects are holding in to each other and are therefore not GCs. Objects that end up on the VMRoot object are static objects which are often a potential source of memory leaks.

Step 1: Find the classes that seemed to leak based on the trending dumps
Step 1: Find the classes that seemed to leak based on the trending dumps
Step 2: Traverse the Referrer or Referees to identify who is holding references to how many objects. VMRoot means global static objects
Step 2: Traverse the Referrer or Referees to identify who is holding references to how many objects. VMRoot means global static objects
Step 3: From a class you can drill to the Instances and from there inspect the object and all its captured primitive values
Step 3: From a class you can drill to the Instances and from there inspect the object and all its captured primitive values

Automatic Out-of-Memory Analysis

Taking memory dumps in production is something most people don’t dare to do as it stalls the JVM for a while (depending on the size of the heap). In case you experience OutOfMemoryExceptions Dynatrace automatically captures a full memory dump as the system is in an unstable state anyway and it is better to capture evidence before it is too late. This feature is only available for Java in the moment.

Found a Memory Leak? Tell us about it

If you found a memory leak in your application let us know. We are happy to also look at your data as part of our Share Your PurePath program.