I keep getting questions about how to best analyze memory leaks – especially when they are not always reproducible by the developer on the local workstation. If you never experienced a memory leak issue (or you simply don’t admit it) then read up on some real life examples on our blog: Fixing Memory Leak in Java Production Environment or DevOps way to solving Memory Issues.
If you want to test this out on your Java or .NET Application follow my steps in this blog, watch my webinar on YouTube where I describe both Memory but also Thread Diagnostics with Dynatrace or read the technical documentation on Memory Diagnostics.
Step 1: Tooling – We use the Dynatrace AppMon Free Trial
Step 2: Connecting your JVM or CLR
The best way is to simply follow the steps in the tool to setup a new Application Environment. There is a step-by-step guide on how to connect an agent into your Java or .NET Application on the Dynatrace Community.
Step 3a: Memory Analysis Server
It is recommended to install and use a Dynatrace Memory Analysis Server. It should be installed by default but may not run automatically. Therefore check your services on windows and make sure you launch the dtanalysisserver on Linux. Before you launch it, you want to edit dtanalysisserver.ini and change the –Xmx to the size of the heap you want to analyze, e.g: -Xmx4G to analyze a 4 GB Heap.
Step 3b: Pre-Requisites for .NET Memory Monitoring
Info: Since Dynatrace 6.1 the option “Automatic” means that the .NET agent tries to turn on the necessary profiler callbacks on the fly and then switches them off after the memory dump is finished so this way the application doesn’t have to be restarted. This temporary switching works only with .NET application from version 4.0 running in blocking Garbage Collector mode.
This step is just for .NET Environments. Due to the nature of .NET there are two things you need to ensure:
- The user that runs your .NET Process is allowed to capture windows performance counters. The user must be in the windows group “Performance Monitor Users”
- You need to enable .NET Memory Dumps in Dynatrace through the System Profile -> Agent Group -> Agent Mapping -> Advanced Dialog
Step 4: Trending Dumps to Detect Problematic Classes
Once you have a Java or .NET agent connected to Dynatrace you can start creating memory dumps. We need to open the Total Memory Dashlet which can be done either through the Cockpit (tree view on the left of the client), Start Center (Start Button in the toolbar – > then select Memory Diagnostics) or through the Process Health dashboards:
Tip: If your button is greyed out it means you don’t yet have an agent connected. Go back to the Application Environment Wizard and walk through connecting your agent
There are two types of dumps: Memory Consumption Trending and Deep Memory Leak Analysis. We start with the trending dumps as the goal here is to create several of these “Lightweight” dumps to figure out which objects from which classes are growing over time:
Tip: I typically uncheck the box “Force GC before creating snapshot”. Why? Because I want to see all objects on the heap which allows you to also identify objects with a high churn rate. With the option checked we would only see objects that are still referenced after a GC run.
Once you have the trending dumps you can either analyzed one of these dumps individually to inspect number of instances per class and also the amount of memory allocated.
But even better is if you select multiple dumps and then watch the trending line to see whether memory is increasing over time:
Tip: If I see a trend like this I select the dump with the lowest and highest usage and click on compare. You can select multiple and compare but we get similar results by comparing the two dumps with the biggest difference
Tip: It doesn’t make sense to focus on String or other basic types. It’s better to look at custom classes or framework classes such as Hibernate, SharePoint, etc., as these classes ultimately reference the primitive types. Start searching in the Dynatrace Client by clicking in the table and start typing e.g: com.mycompany. All views have this built-in filtering capability.
Step 5: Detecting Memory Leak Root Cause
Once you have identified classes through the trending dumps that you believe shouldn’t grew that much in size and count it is time to do a deep dive memory leak analysis. For this we create another dump – this time selecting the Deep Memory Leak Analysis option – optionally with capturing Strings (Java Only) and Primitive values:
Tip: Capturing such a memory dump may take up to several minutes as lots of data is getting captured. In production environments this has to be used with care.
Now it’s time to analyze the captured instances – focusing on those classes we identified through the trending dumps. The following screenshots show you how to walk the referrer trees in order to figure out which objects are holding in to each other and are therefore not GCs. Objects that end up on the VMRoot object are static objects which are often a potential source of memory leaks.
Automatic Out-of-Memory Analysis
Taking memory dumps in production is something most people don’t dare to do as it stalls the JVM for a while (depending on the size of the heap). In case you experience OutOfMemoryExceptions Dynatrace automatically captures a full memory dump as the system is in an unstable state anyway and it is better to capture evidence before it is too late. This feature is only available for Java in the moment.
Found a Memory Leak? Tell us about it
If you found a memory leak in your application let us know. We are happy to also look at your data as part of our Share Your PurePath program.