Are you prematurely restarting JVMs to keep them from running out of memory? Or have you received the recommendation from your application experts to just “increase the Java Heap Space?”
The following memory chart shows the memory consumption of 10 JVMs (4.1GB Heap each) per Host on an 8 machine cluster. The JVMs kept running out of memory, crashing with Out Of Memory Exceptions and sometimes even bringing the whole host to crash:
The typical first response from application teams in situations like this is to add more memory to these hosts. However – as this is not the best practice to solve problems – the team where this story is from decided to take a more pro-active approach and decided that it was time to look closer on the actual root cause of the exhaustive memory consumption.
Step #1: Analyzing the Out-of-Memory Dumps
Whenever the team’s JVMs crash, its APM Solution actively captures a full heap memory dump. This makes it very convenient to analyze post mortem the root cause of the excessive memory usage. In this scenario the hotspot was easy to identify. The following screenshot shows that the root causes are 22.7k T4CStatement objects. They consume about 2.6GB on the heap and the reason why they are not cleared by the Garbage Collector is because they are referenced on global static variables:
Step #2: Who is allocating these objects and why are they not cleaned
Taking a closer look on who allocates these T4CStatement objects shows that they are allocated whenever a SQL Statement is executed through the allocateStatement method called by createStatement. The problem with that method is that the T4CStatement object is allocated but it is never freed when the associated connection gets put pack into the Connection Pool:
Not releasing them keeps them on the heap until all the memory is consumed. The following screenshot shows that most of these objects have been on the heap for hours or even days:
Step #3: Working with Vendor to fix the problem
Oracle, the vendor of the JDBC Driver, dug into its implementation and found where it keeps these statement objects in a global variable so that it can’t be cleaned by the GC. With a quick turnaround the vendor provided a fix so that these T4CStatement were no longer referenced longer than these transactions needed them. That solved all crashes and of course reduced average memory consumption. That also leads to overall more efficient use of their JVMs in the cluster and processing more load on the same environment without fearing to run into memory issues.
Learn more on Enterprise Memory Management?
I can also recommend checking out our free online book chapter on Memory Management. Or read up on some of the other memory management & memory leak related blog posts such as DevOps way to solving JVM Memory Issues, Fixing Memory Leaks in Java Production Applications or Top Java Memory Problems.