Memory Management is a tough topic in managed runtime environments like .NET or Java. Checking the available performance counters is a good start to identify memory leaks or memory problems. The .NET Runtime exposes several interesting counters that should be monitored:
- Gen0, Gen1, Gen2 and Large Object Heap Size
- Gen0, Gen1 Promoted Bytes/Sec
- Promoted Memory from Gen0, Gen1
- Total commited and reserved Bytes
Using a Performance Counter Monitoring tool like Windows System Monitor (Control Panel->Administrative Tools->Performance) gives you a good live overview of your processes memory. The following image shows the Gen0 (green), 1 (yellow), 2 (orange) and Large Object Heap Size (red) of two processes on my system. The two processes are two instances of an ASP.NET Worker Process. One is hosting the frontend ASP.NET Web Application, the other is hosting the backend ASP.NET Web Services.
Looking at the Dynatrace memory graph above shows me that I seem to have a memory problem on my frontend process. I trusted those values and spent quite some time to figure out what the memory leak actually is creating different memory dumps to identify the growing objects by comparing the individual dumps. Interestingly enough – the dumps of my frontend component didn’t indicate any problem. Just out of curiosity I did dumps on my backend process and was surprised to find a huge leak there.
.NET Heap Counters are not always accurate
When having more than one process instance of the same application .NET Heap Counters are not always accurate. Here is why:
Additionally to the Heap Counters I also monitored the general process performance counters. Here is what the counters looked like for my frontend and backend process:
These counters backed up the information I got from my memory dumps indicating the memory leak is actually in my backend process and not in the frontend. The frontend looks really fine with a flat line in the memory consumption.
What seems to be the problem?
In order to rule out a tool problem I used different tools for the performance counters. All tools showed the same inconsistency between Heap Counters and Memory Counters. It seems that this problem happens if you have more than one instance of the same application. In that case – the instance name of the process for the performance counters get an index appended. The first process would be “aspnet_wp.exe”, the second would be “aspnet_wp.exe#2” and so on. It seems that there is a problem in the .NET Framework publishing the Heap Counters to the correct object instance.
Performance Counters for Memory analysis in .NET Applications are a great source to identify memory problems. Be careful though when you run the same application multiple times on your machine. Especially when running ASP.NET Apps on IIS in different Application Pools. The above explained behaviour might not occur on every machine – I am sure it depends on the sequence that processes are started. I am going to forward this information to the folks at Microsoft.