NAM Probe in a VMware environment

Virtualization and VMware support

We have two choices regarding the monitoring network traffic between VMs on the same ESX host:

  • Run NAM Probe on a VM and have it monitor traffic on the various vSwitches on the host.
  • Run NAM Probe on a physical machine and tap or span the traffic off of the vSwitches.

Each represents different challenges.

Virtual NAM Probe

It is reasonable to assume that a Virtual NAM Probe implementation will not be able to command the same set of compute resources as a physical NAM Probe and therefore cannot be expected to perform as well or do as much.

One thought put forward is that the focus of a Virtual NAM Probe should be limited to monitoring only traffic between VMs local to the ESX host. Traffic that comes in to, or leaves the host, should be ignored and picked up by a physical NAM Probe.

This can become problematic when one of the VMs that is being monitored gets vMotioned off to another ESX host. Traffic that was destined to, or received by, that VM is no longer local to that NAM Probe and needs to be filtered out.

Likewise, the monitoring NAM Probe on the ESX host that received the new VM would need to recognize it as being local.

To accomplish this task, the NAM Server and Virtual NAM Probe must be made VMware aware in order to determine which VMs are local to the ESX host and the NAM Probe filters need to be automatically configured accordingly.

Some of the awareness may be provided via the vCenter interfaces used in Server Vantage and Business Service Management (BSM), however, in cases where just Network Application Monitoring is utilized, the NAM Server will need the added capability of querying the vCenter for the information. It is also possible that the NAM Probe could be made to query its ESX host for local VM information and self configure accordingly.

Port mirroring and tapping

The current accepted method of port mirroring traffic from an ESX host is through the use of Cisco's Nexus 1000V virtual distributed switch. This method imposes additional licensing costs and requires an external switch or router (preferably Cisco) to terminate the span. This, in turn, may create possible ownership contention between network and server groups and a possible commitment to use additional Cisco technology.

Also available are commercial virtual taps such as the Net Optics phantom tap, which operates independently of the vSwitch and works at the hypervisor level. It is designed to deliver packets via a GRE tunnel to an external Net Optics “Data Director” data monitoring switch, which adds to the cost of this solution. There are no known technical reasons why this solution should not function correctly. However, it has not been certified by Dynatrace and therefore Dynatrace Customer Support might have limited means of assisting you in troubleshooting such an environment.

VLAN monitoring

Because the virtual standard switch cannot mirror the traffic data, you should use the promiscuous mode option. When configuring the standard portgroup for the virtual standard switch, setting the VLANID option to 4095 will allow the virtual adapter to monitor all traffic across all VLANs.

On the virtual distributed switch, however, setting the VLANID option to 4095 is not available. If VLAN tagging is not in use, the VLANID can be set to none. If VLAN tagging is in use, the VLANID used in configuring the distributed portgroup must match. Monitoring traffic from multiple VLANs requires a portgroup configuration for each VLAN being monitored.

VMware vSphere 5.x

There are two basic vSphere switch types available in a vSphere environment. The vSphere standard switches handle network traffic at the host level and the vSphere distributed switch functions as a single switch across all associated hosts.

You can create a vSphere distributed switch on a vCenter Server datacenter. After you have created a vSphere distributed switch, you can add hosts, create distributed port groups, and edit distributed switch properties and policies.

Port mirroring is the ability of a network switch to copy network packets seen on a switch port and send duplicates to a network monitoring device. In VMware vSphere 5, a distributed switch provides a similar ability. After a port mirror session is configured with a destination (a virtual machine, vmknic, or uplink port), the distributed switch send copies of packets to the destination.

Port mirroring in a vSphere 5.5 environment

(source: Create a Port Mirroring Session with the vSphere Web Client)

To configure port mirroring in vSphere 5.5 environment:

  1. Browse to a distributed switch in the vSphere Web Client navigator.
  2. Click the Manage tab and select Settings > Port Mirroring
  3. Click New and select the session type for the port mirroring session.
header header
Distributed Port Mirroring Mirror packets from a number of distributed ports to other distributed ports on the same host. If the source and the destination are on different hosts, this session type does not function.
Remote Mirroring Source Mirror packets from a number of distributed ports to specific uplink ports on the corresponding host.
Remote Mirroring Destination Mirror packets from a number of VLANs to distributed ports.
Encapsulated Remote Mirroring (L3) Source Mirror packets from a number of distributed ports to remote agent’s IP addresses. The virtual machine’s traffic is mirrored to a remote physical destination through an IP tunnel.
Distributed Port Mirroring (legacy) Mirror packets from a number of distributed ports to a number of distributed ports and/or uplink ports on the corresponding host.
  1. Set the session properties. Different options are available for configuration depending on which session type you selected.
header header
Name You can enter a unique name for the port mirroring session, or accept the automatically generated session name.
Status Use the drop down menu to enable or disable the session.
Session type Displays the type of session you selected.
Normal I/O on destination ports Use the drop-down menu to allow or disallow normal I/O on destination ports. This property is only available for uplink and distributed port destinations. If you disallow this option, mirrored traffic will be allowed out on destination ports, but no traffic will be allowed in.
Mirrored packet length (Bytes) Use the check box to enable mirrored packet length in bytes. This puts a limit on the size of mirrored frames. If this option is selected, all mirrored frames are truncated to the specified length.
Sampling rate Select the rate at which packets are sampled. This is enabled by default for all port mirroring sessions except legacy sessions.
Description You have the option to enter a description of the port mirroring session configuration.
  1. Select the source of the traffic to be mirrored and the traffic direction. Depending on the type of port mirroring session you selected, different options are available for configuration.
header header
Add existing ports from a list Click Select distributed ports. A dialog box displays a list of existing ports. Select the check box next to the distributed port and click OK. You can choose more than one distributed port.
Add existing ports by port number Click Add distributed ports, enter the port number and click OK.
Set the traffic direction After adding ports, select the port in the list and click the ingress, egress, or ingress/egress button. Your choice appears in the Traffic Direction column.
Specify the source VLAN If you selected a Remote Mirroring Destination sessions type, you must specify the source VLAN. Click Add to add a VLAN ID. Edit the ID by using the up and down arrows, or clicking in the field and entering the VLAN ID manually.
  1. Select the destination for the port mirroring session. Depending on which type of session you chose, different options are available.
header header
Select a destination distributed port Click Select distributed ports to select ports from a list, or click Add distributed ports to add ports by port number. You can add more than one distributed port.
Select an uplink Select an available uplink from the list and click Add to add the uplink to the port mirroring session. You can select more than one uplink.
Select ports or uplinks Click Select distributed ports to select ports from a list, or click Add distributed ports to add ports by port number. You can add more than one distributed port. Click Add uplinks to add uplinks as the destination. Select uplinks from the list and click OK.
Specify IP address Click Add. A new list entry is created. Select the entry and either click Edit to enter the IP address, or click directly in the IP Address field and type the IP address. A warning appears if the IP address is invalid.
  1. Review the information that you entered for the port mirroring session on the Ready to complete page. The new port mirroring session appears in the Port Mirroring section of the Settings tab.

Port mirroring in a vSphere 5 environment

To configure port mirroring in vSphere 5 environment:

  1. Start the Create Port Mirroring Session configuration wizard.

  2. On the General Properties screen, configure basic settings and then click Next.

  • In the General section, type a name and description for the port mirror.

  • In the Port Mirroring Session Details section:

Select the Allow normal IO on destination ports check box if you want to enable normal I/O on the destination port.

Select the Encapsulation VLAN check box if you want to have a VLAN encapsulate the mirrored packets.

  1. On the Specify Sources screen, specify the source that you want to monitor and then click Next.
  • On the Traffic direction menu, choose Ingress, Egress, or Ingress/Egress, to describe the traffic you want to monitor.

  • Specify the Port ID of that source virtual machine.

Note

To determine the corresponding dvPort number or Port ID number of a virtual machine:

  1. Switch to HomeInventoryNetworking.

  2. Select dvSwitch.

  3. Click the Ports tab on the right panel and scroll down to see virtual machines and associated port IDs.

  4. Enter the port number in the Port ID field and move it to the right panel.

  1. On the Specify Destination screen, specify where you want to send the mirrored traffic.

On the Destination type menu, select Port or Uplink.

  1. To enable the port mirroring session, click Edit.

In the Edit Port Mirroring Session dialog box, set Status to Enabled.

Cisco Nexus ERSPAN

Copy data from a Cisco virtual switch (for example, Nexus 1000v) to an external physical device using ERSPAN (see below).

The example and configuration details are provided here only as reference building blocks, as an indication of what is required, and should not be considered the definitive configuration and syntax requirements for all situations.

Encapsulated remote switched port analyzer (ERSPAN)

ERSPAN transports mirrored traffic over an IP network. The traffic is encapsulated at the source router and transferred across the network. ERSPAN adds a GRE header and an ERSPAN header (both 8 bytes), so the destination needs to be able to handle both of these. At the destination switch, both of those headers are stripped and the base packets are forwarded out the destination interface.

ERSPAN consists of an ERSPAN source session, routable ERSPAN generic routing encapsulation (GRE)-encapsulated traffic, and an ERSPAN destination session. You must separately configure ERSPAN source sessions and destination sessions on different switches.

ERSPAN configuration requirements

Configuration of ERSPAN is a multistep process in which the ERSPAN source and ERSPAN destination are defined. The first step for ERSPAN on a Nexus 1000v is to configure the ERSPAN port profile. Port profiles are used to configure interfaces on the Virtual Ether Module (VEM).

N1000v-VSM# conf t
N1000v-VSM (config-port-prof)# port-profile N1KV-ERSPAN

Indicate that the interface created will use Layer 3 (IP) control functions.

N1000v-VSM (config-port-prof)# capability l3control
N1000v-VSM (config-port-prof)# vmware port-group
N1000v-VSM (config-port-prof)# switchport mode access
N1000v-VSM (config-port-prof)# switchport access vlan 702
N1000v-VSM (config-port-prof)# no shutdown

Indicate a VLAN that will be brought up before communication between the VEM and VSM is established.

N1000v-VSM (config-port-prof)# system vlan 702
N1000v-VSM (config-port-prof)# state enabled
N1000v-VSM (config-port-prof)# end

The two differences between the ERSPAN port profile and a port profile that would be used directly by a guest machine are the following commands:

N1000v-VSM (config-port-prof)# capability l3control
N1000v-VSM (config-port-prof)# system vlan 702

The capability l3control command indicates that the interface created will use Layer 3 (IP) control functions. In this case, ERSPAN will encapsulate traffic in GRE for transport across an IP network.

The system vlan 702 command indicates a VLAN that will be brought up before communication between the VEM and Virtual Supervisor Module (VSM) is established.

Next, use the ERSPAN port profile to create a new vmk port on the host from which traffic will be sourced. This includes the assignment of a unique IP address for the ERSPAN vmk to use, the subnet mask, and the default gateway.

Having created the port profile and successfully had that assigned to a vmk port, you can now configure the ERSPAN source and destination parameters on the appropriate devices.

For simplification in our example, there is a single VMware ESX host with the Nexus 1000v installed, and we are looking to mirror traffic from a single Virtual Ethernet Interface via a physical external switch that is one hop from the virtual environment with our probe connected to a Gigabit interface on the physical switch.

Figure 1. Simplified ERSPAN network diagram

Simplified ERSPAN network diagram
Simplified ERSPAN network diagram

Source and destination IP addresses are configured under the ERSPAN monitor session along with the ERSPAN ID. ERSPAN IDs are important in that they uniquely identify an ERSPAN session and can be used to differentiate between multiple ERSPAN sessions at the destination. This is particularly helpful in a centralized model where there might be many ERSPAN sessions aggregated. In our example we need to first configure the source switch, including the mirror source (Virtual Ethernet 27), the GRE tunnel destination (for example, a local interface on the physical switch 10.1.1.100), and the ERSPAN ID (in our example, 1000).

N1000v-VSM # conf t
N1000v-VSM (config)# monitor session 1 type erspan-source
N1000v-VSM (config-erspan-src)# description ERSPAN Session 1

Provide the mirror source.

N1000v-VSM (config-erspan-src)# source interface vethernet27

Specify the IP address of the ERSPAN destination.

N1000v-VSM (config-erspan-src)# destination ip 10.1.1.100

Indicate the ERSPAN ID used to differentiate between aggregated sessions.

N1000v-VSM (config-erspan-src)# erspan-id 1000

ERSPAN sessions are configured in shutdown by default, so they need to be enabled.

N1000v-VSM (config-erspan-src)# no shut
N1000v-VSM (config-erspan-src)# end

Having configured the source of the ERSPAN, the next stage is to configure the other end of the session, which in our example is the physical switch. In addition to providing a termination point to the ERSPAN GRE tunnel, we also need to provide an output (destination) for the mirrored traffic, which in our example is the Gigabit interface 2/2, where the mirrored traffic should be presented stripped of both GRE and ERSPAN headers.

Physical_switch # conf t
Physical_switch(config)# monitor session 1 type erspan-destination

Provide the destination for the mirrored traffic

Physical_switch (config-erspan-dst)# destination interface Gi2/2

Specify the IP address of the ERSPAN destination (the source switch)

Physical_switch (config-mon-erspan-dst)# source
Physical_switch (config-erspan-dst-src)# destination ip 131.176.62.38

Provide the ERSPAN ID used to differentiate between aggregated sessions

 Physical_switch (config-erspan-dst-src)# erspan-id 1000

ERSPAN sessions are configured in shutdown by default, so they need to be enabled.

Physical_switch (config-erspan-dst-src)# no shut
Physical_switch (config-erspan-dst-src)# end

Verification

The monitor session's configuration and status can be reviewed and verified with the show monitor command; the following screen shows the expected output for our example configuration.

N100v-VSM(config-erspan-src)# show monitor session 1
session 1
---------------
description				: ERSPAN Session
type					: erspan-source
state					: up
source intf				:
				rx		: Veth27
				tx		: Veth27
				both	: Veth27
source VLANs			:
				rx		:
				tx		:
				both	:
Filter VLANs			: filter not specified
destination IP 			: 10.1.1.100
ERSPAN ID				: 1000
ERSPAN TTL				: 64
ERSPAN IP Prec.			: 0
ERSPAN DSCP				: 0
ERSPAN MTU				: 1500

Caution

A number of restrictions apply to what can be used as source and destination ports in an ERSPAN configuration, and these often vary for different architectures and OS versions, so it is suggested that you refer to Cisco for details relevant to the specific environment. In addition to those limitations, it is important to be aware of other issues that may affect the successful implementation of ERSPAN.

  • The ERSPAN tunnel contains both GRE header and ERSPAN header and the destination device must be capable of understanding and be able to terminate the tunnel appropriately.

  • In our example, the physical switch was only a single hop away from the virtual environment, but it is possible to route ERSPAN traffic across and via multiple devices, and any interim device in the path, although not terminating the ERSPAN tunnel, needs to be able to support ERSPAN in order to pass it.

Additional references

vSphere Networking

Vsphere ESXI Vcenter Server 55 Networking Guide

Cisco Nexus 1000V System Management Configuration Guide, Release 4.0(4)SV1(3)

Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.0(4)SV1(3)

Cisco Nexus 1000V Interface Configuration Guide, Release 4.0(4)SV1(3)

Cisco Nexus 1000V Command Reference, Release 4.0(4)SV1(3)