VMware

Overview

To run your application in a virtual environment, you must have a virtualization-aware monitoring tool. Use AppMon to monitor the whole stack, from the application down to the guest OS and to the virtual machine and the physical host.

This page describes how to leverage AppMon's monitoring capability for VMware. In addition, it includes special requirements for running AppMon in a virtual machine.

About VMware monitoring

AppMon's VMware monitoring uses the vSphere Management SDK for vSphere  (VMware Infrastructure (VI) SDK for older versions). See System requirements for supported vSphere versions.

Note

vSphere 6.x monitoring is currently not supported.

Monitoring with predefined monitors and dashboard

AppMon includes a predefined dashboard for VMware monitoring. In the AppMon Client, select Dashboard > Open and select VMware - Monitoring to open the VMware Monitoring dashboard.

AppMon automatically installs two monitors for VM monitoring when you set up system monitoring. One is for the VMware virtual machine performance monitor, the other is for the VMware Host system performance monitor. These are named Virtual Machine and VM Host Systems respectively. This page describes how to use the VMware - Monitoring dashboard and adjust configuration for these monitors.

To use the VMware - Monitoring dashboard:

  1. Click the Edit Filter button in the AppMon Client toolbar to open the Dashboard Filter dialog box.
  2. Click the Hosts tab, then click the Selected radio button make sure Hosts is selected in the list box.
  3. Select the filters for the physical host and the virtual machine(s) that you want to monitor. For example, for a virtual machine called dtserv running on a physical host called ESXiDemo3, select those hosts. Click OK.

Host filtering on the VMware - Monitoring dashboard

Monitoring virtual hosts

Use the VMware Host System Performance Monitor to monitor the physical machines that host your virtual machines.

  1. In the AppMon Client, double-click the desired system profile for VMware monitoring and select Monitors.
  2. Under VMware Host System Performance Monitor, select VM Host Systems and click Edit.
  3. In the Scheduled Monitor Editor dialog box, enter a username and password for the VMware host account.
  4. Decide whether to use a vCenter facility to monitor all your hosts, or monitor each one individually.
    • If you use vCenter, use the vCenter server host name (https://<vCenterHostName>/sdk/vimService) or the variable host as the VMware web service URL value and click OK. Your URL should look like the following, regardless of how many, and which hosts you monitor: https://<host>/sdk/vimService.
  • If you want to monitor hosts individually, click the Add Hosts button to add them to the Host section at the bottom of the Scheduled Monitor Editor dialog box. AppMon comes with a predefined VMHost host group. Use the host group in the Monitor and add hosts to that host group using the infrastructure management facility.

Now you can activate the Monitor and get periodic measures for all of your vHosts. VMware typically has a granularity of 20 seconds, so it does not make sense to have a schedule lower than this. Scheduling once per minute should be sufficient.

Monitoring virtual machines

If you use vCenter, simply monitoring the virtual hosts to monitor virtual machines. Use the vCenter host in the URL, as described in Monitoring Virtual Hosts. Add the hosts that represent the virtual machines to the host section.

AppMon monitors all hosts designated as virtual machines and gets respective performance counters from vCenter. Use the predefined VMware host group in the Monitor and add virtual machines to that host group.

If you can't use vCenter to monitor all hosts, define a separate monitor for each physical vHost that hosts virtual machines. In this case, configure the URL with the real host name of the physical host. Add the virtual machines that are hosted to the host section.

Notes

  • You must install guest tools for VMware Management to know the virtual machine host name. Without guest tools installed, you must configure the virtual machine name and use a separate Monitor for each virtual machine.

  • If you have guest tools installed, never use the Virtual Machine parameter. Instead, use the host section to identify the virtual machines to monitor.

Interpreting VMware measures

The following links show performance counters and categories for each counter data object type (VMware web site):

See Datacenter Administrator Guide for more information on troubleshooting.

Applications in a virtual machine

A virtual machine does not run exclusively on a host. It has to share its resources with other virtual machines. This may cause the virtual machine to suspend at any time. The guest OS is usually not aware of this which can cause a timekeeping problem.

As a result, the timings captured in a virtual machine are distorted. The extent of the distortion depends on the following in the guest operating system:

  • Response times may appear to be less than they actually are. This is called apparent time and means that the time that has passed for the system is less than in reality because it was suspended in between.
  • For the same reason, CPU Times might be more or less than they actually are. This is a problem particularly on older Linux and Unix systems; and nearly all Windows systems. This is due to tick based timing.

To fix this, do one of the following:

  1. Use an operating system with a tick-less timer. Nearly all new Linux and Unix Systems use tick-less timers. This reduces both timing problems. See the associated documentation for more information.
  2. Use the pmc0x1001 Agent Timer. This instructs AppMon to capture the real elapsed time from the VMware infrastructure. This leads to more accurate response times at the cost of a higher overhead. Look in the Agent log from the System Information dashlet to see the amount of overhead.

The default timer:

2010-10-13 09:26:56 info    [java  ] Calibrating overhead calculations...
2010-10-13 09:26:59 info    [java  ] Reference method has a mean CPU execution time of 2179ns
2010-10-13 09:26:59 info    [java  ] Getting CPU time of current thread costs 1714ns

The PMC timer:

2010-10-13 09:28:09 info    [java  ] Calibrating overhead calculations...
2010-10-13 09:28:12 info    [java  ] Reference method has a mean CPU execution time of 4477ns
2010-10-13 09:28:12 info    [java  ] Getting CPU time of current thread costs 1723ns

Both of these logs are from the same virtual machine and Java application. The overhead per instrumented method increased from roughly 2000ns to ~4500ns because of the call to the hypervisor.

Note

This demonstrates that the overhead of a PMC timer is higher than the default one. The timings here are not a general baseline for AppMon overhead. This is very specific to the underlying hardware and guest operating system, and was taken in a VM.

To use the PMC Agent timer, you must activate the feature for the virtual machine itself. These special timers must be enabled on the guest instance by setting monitor_control.pseudo_perfctr = TRUE for the instance. Depending on the VMware product you use, do this in  the management interface or manually edit the instance's .vmx file.

Supported systems include VMware ESX and ESXi 3.5+, VMware Server 2.0+ and VMware Workstation 6.5+. Other systems may support these timers as well.

Running AppMon in a virtual environment

In addition to the items in the Deployment Guide, there are requirements to successfully run AppMon in a virtual environment.

AppMon Server in a virtual machine

The AppMon Server is very demanding on the hardware and periphery. It operates well within a virtualized environment if you follow these rules:

  • Memory
    VMware may cause you to over-commit on memory. This means that the sum of memory assigned to virtual machines can exceed the physical memory available on the host. This is managed using memory ballooning. The result is that a virtual machine may not be able to use its assigned memory and may fall back to swapping.
    Swapping is devastating. You must ensure that the AppMon Server itself is never swapped. To ensure that, you must explicitly reserve enough memory for the virtual machine to accommodate for the whole AppMon Server, as well as the necessary parts of the guest OS in physical memory. You can verify this by monitoring the ballooning memory of the AppMon Server's VM. There must be no ballooning.
  • Virtual CPUs
    VMware generally states that you should run VMs with a single virtual CPU, unless the running software takes special advantage from multiple CPUs. AppMon does take advantage of multiple CPUs and cores. You should configure the VM with multiple virtual CPUs. You get the best results if the number of virtual CPUs and cores equals the number of physical cores.
  • Disk
    AppMon writes a lot of data to disk if you use session recording. Make sure that the physical disk and the interface underlying the assigned virtual disk can accommodate the load. In addition to the normal self-monitoring, you must monitor the AppMon Server performance from the VMware - Monitoring dashboard. Ensure that there is no memory ballooning and that the CPU ready time does not exceed 1000ms. (1 second per 20s interval).

AppMon Collector in a virtual machine

The Collector has similar demands on the virtual environment as the AppMon Server, but does not use a lot of disk. Make sure you meet the demands of CPU and network memory.