Virtualization requirements and best practices

This page provides a requirements and best practice guide for running an AppMon server virtualized. AppMon can run virtualized, but there are some requirements to guarantee a smooth running server.

Dedicated resources

Dynatrace recommends dedicated or entitled cores, memory, and disk for the server using a VMWare resource pool. Overcommitment of cores or memory can cause performance problems during data collection and force the server to skip some data such as PurePaths.

Virtual CPUs

For most processes, VMWare recommends a single virtual CPU for a virtual machine (VM). This reduces the management overhead on both the guest system and VMWare. A single virtual CPU however has the same impact as a single physical CPU. Only a single thread at a time can be executed. The AppMon Server has a lot of threads and it must therefore be an exception to this rule. In addition, VMWare cannot parallelize something that is not executed in parallel by the VM. That means a VM with two virtual CPUs can never use more than two physical CPUs at the same time. In that regard, AppMon depends on the physical hardware power. As a general rule, the number of vCPUs depends on the size of the setup. You should take the number of cores from the Default Deployment Sizes as a base recommendation. If the numbers do not fulfill the requirements, then increase the cores on the VM.

Memory

VMWare does resource pooling on available physical memory if not otherwise instructed. That means you can have four VMs with each having 4 GB memory and physical hardware with only 12GB. Also consider that the hypervisor needs memory (approx. 500 MB).

The vHost can instruct the VM not to use a portion of the assigned memory. This is known as ballooning, and it lets VMWare dynamically assign available memory from one VM to the other. This can cause the VM to swap out running processes or parts of them.

Swapping in general can harm application performance. It is particularly harmful to java applications because memory is cycled through all the time, which is harmful to the AppMon Server. Thus, the whole of a java program must always be in physical memory. As a rule, the VM running the AppMon Server must have enough reserved memory to accommodate the AppMon Server plus the operating system. See Default Deployment Sizes for proper memory sizing.

I/O and network

The AppMon Server writes a lot of data to disk for session recording, receives a lot of data, and also sends a lot data to the database. Disk and network are still shared resources in a VM, so ensure there are enough resources reserved to the AppMon Server VM.

Guidelines

  • Separate Network Switch for incoming traffic to the AppMon Server. See the bandwidth requirements in Network Bandwidth Best Practices to learn how.
  • Make sure that all other traffic (database, client, disk) is taking a different physical network switch.
  • Disks are mostly connected from SAN to a virtual machine. Make sure the network switch can take the load and, if possible, is dedicated to the usage of the SAN.

See Disk IO Determination Best Practices for more guidance regarding disk I/O for deployment, See Network Bandwidth Best Practices for more guidance regarding network load.

Understanding VMware performance charts

As an VMWare hypervisor administrator, you must understand and know your infrastructure. The following table shows performance chartsidentify, troubleshoot, and resolve issues.

Key CPU metrics

Counter name in API Measurement Unit Description Desired value
cpu.ready,summation Ready ms Ready time is the time spend waiting for any CPUs to become available in the past update interval. Be aware that this is a counter and is dependent on a time interval, such as 20 seconds. Should be close to 0.
cpu.idle.summation Idle ms CPU Idle  
cpu.used.summation Used ms CPU Used  
cpu.swapwait.summation Swap Wait ms Swap wait time is time spent waiting for memory to be swapped in. When the VM is waiting for memory, it is not doing work. Should be 0.

Key memory metrics

Counter name in API Measurement Unit Description  
mem.vmmemctl.average Balloon kB The amount of memory currently claimed by the balloon driver. This does not represent a performance problem, but indicates the host starting to take memory from less needful VMs for those with large amounts of active memory. But if the host is ballooning, check swap rates (swapin and swapout) which would be indicative of performance problems. Should be 0.
mem.swapin.average Swap in kB The rate at which memory is being swapped in from disk. A larger number indicates lack of memory and is a clear indication that performance is suffering as a result. Should be 0.
mem.swapout.average Swap out kB The rate at which memory is being swapped out to disk. A larger number indicates lack of memory and is a clear indication that performance is suffering as a result. Should be 0.
mem.swapinrate.average Swap in rate kB/s The swap in rate reports the rate at which a VM's memory is swapped in from disk. Should be 0.
mem.swapoutrate.average Swap out rate kB/s The swap out rate reports the rate at which a VM's memory is swapped out to disk.  Should be 0.
mem.swapped.average Swapped kB Memory Swapped (Average). Should be 0.
cpu.swapwait.summation Swap Wait ms Swap wait time is time spent waiting for memory to be swapped in. When the VM is waiting for memory, it is not doing work. Should be 0.

See VMWare performance counters for more information.