When your application experiences memory leaks or high object churn it’s important that you get your hands on the memory dumps so you can analyze these issues. In production environments, this is often quite a challenge as you most likely can’t log into the environment, and you may have no other means of triggering memory dumps. Dynatrace now enables you to both trigger and securely download memory dumps to subsequently analyze them in your analysis tool of choice.

Triggering a memory dump

To trigger a memory dump

  1. Navigate to the Host page, Service page, Process page, or Process group page of the entity that you are that has issues.
  2. Click the Browse […] button and select Memory dump details.
    The Memory dumps page displays a list of the memory dumps that occurred during the selected time frame for the selected entity (host, service, process, or process group).
  3. Click the Trigger new dump button to generate a new memory dump (you’ll need the Configure capture of sensitive data user permission to do this. See permissions for viewing confidential data for more details). To trigger a memory dump first select the process. Note: Dynatrace currently supports memory dumps for Java and Node.js processes only.
  4. Click the Trigger dump button. 
    It takes a few minutes to generate a memory dump. The time required varies widely based on application type. Java applications that have multiple GBs of heap memory may take several minutes; smaller dumps will be available almost immediately.
  5. Return to the Memory dumps page. The newly created dump now appears in the list.
  6. Click the memory dump Download link to view the memory dump (see example below).
    Memory dumps are first stored locally on the disk of the monitored application-server machine. The Dynatrace OneAgent can subsequently upload it automatically to a Security Gateway which can act as a long-term storage center for memory dumps. The provided download links direct you to the Security Gateway received that memory dump. This approach ensures that the download is only available to users who have access to the network location of your Security Gateway. This precaution provides an additional security layer that ensures no sensitive data leaves your data center without you explicitly configuring it that way.

  7. In case of Java applications, the download provides the memory dump in hprof format, which can be analyzed using a number of tools, including Eclipse Memory analyzer and VisualVM (see example below).

    The process works the same for Node.js applications. Node.js memory dumps can be opened in Google Chrome’s integrated memory heap snapshot analysis tool (see example below).

Configure a Security Gateway for memory dump storage

Note: If you don’t already have a Dynatrace Security Gateway setup, you need to install a Security Gateway.

  1. Access the config.properties file of your Security Gateway to make a few modifications. This file is located in the config directory of your Security Gateway:
    – Linux filepath: /var/lib/dynatrace/gateway/config
    – Windows filepath: %ProgramData%\dynatrace\gateway\config
    Note: On Windows, the programdata directory is usually hidden by default. Paste the file path above in Windows Explorer to access the file.
  2. Add the following code to the config.properties file and adjust the values based on your requirements.
    [collector]
    ...
    # collector supports collecting of memory dumps from agents (default: false)
    # Add this to the [collector] section
    DumpSupported = true
    ...
    
    # Add this section to the end of the file
    [dump]
    # relative to security gateway installation directory or absolute path
    # The Security Gateway runs with reduced permissions to the file system. Ensure that the user that runs the Security gateway has write permissions to this directory
    dumpDir = dump
    # maximum size in gb for stored memory dump
    maxSizeGb = 100
    # maximum age in days for keeping stored memory dumps
    maxAgeDays = 7
    # maximum number of concurrent file uploads supported
    maxConcurrentUploads = 5
  3. The configured directory must be created manually and the permissions must be set so that the user who runs the Security Gateway has read and write permissions.
    – On Linux, this requires that the user has the Read & execute permission on the directory.
    – On Windows, this requires that the user group Users have the Read, Write, and Modify permissions.
  4. Restart the Security Gateway. All instances of Dynatrace OneAgent will automatically reconnect and recognize that the Gateway can now store memory dumps. Going forward, whenever a memory dump is successfully triggered, it will subsequently be uploaded automatically to the correct Security Gateway. Once safely stored on the Security Gateway, OneAgent deletes the local file to clean up storage.

Note: This feature requires OneAgent 1.127 or higher and Security Gateway 1.127 or higher.

Limitations: 

  • Memory dumps are not available for IBM Java.
  • Memory dumps are not yet available for PaaS integrations.
  • Your application server must have adequate space available to store the heap dump.