How to retrieve performance metrics with Windows Performance Monitor

Goal of this walkthrough

The Microsoft Windows Performance Monitor (PerfMon) moves performance data from Windows based machines into a monitoring environment. To collect this data in AppMon, you’ll do some configuration that affects your Windows environment and the AppMon Collector.

This walkthrough describes how to set up and configure users, permissions, services, and the AppMon Collector. It also covers security on hardening the monitoring solution.

Scenario

The following information has been tested on Windows Server 2003 and 2008 (R2), but should work for other versions. It covers configuring the monitored machine and the AppMon collector. You’ll configure the operation system, so you should have administrative rights and some operation knowledge.

Note

Execute the Windows Performance Monitor on a Windows machine. If it is configured to monitor a remote machine, it still depends on the Windows API to query these metrics.

Prerequisites

Required services

Run the following services on the monitored machine:

Service Service Name
Remote Registry Service RemoteRegistry
Remote Procedure Call (RPC) RpcSs

Do a local check to make sure those services are running and accessible. To validate, execute the following commands:

RemoteRegistry

C:\>sc query RemoteRegistry

SERVICE_NAME: RemoteRegistry
        TYPE               : 20  WIN32_SHARE_PROCESS
        STATE              : 4  RUNNING
                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

Remote Procedure Call (RPC)

c:\>sc query RpcSs

SERVICE_NAME: RpcSs
        TYPE               : 20  WIN32_SHARE_PROCESS
        STATE              : 4  RUNNING
                                (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

Both services should return the RUNNING state.

Firewall settings

To access the previously listed services using a network, make sure that a local firewall does not block traffic. In Windows 2008 the Advanced Firewall does not allow general access, by default. If the monitored machine is a member of an active directory domain, the firewall may be enforced through group policies. Also if the monitoring machine  (AppMon Collector) is in a different domain, no domain or subnet, firewall rules may block the access.

In Windows 2008, enable the predefined Remote Service Management (RPC) rule. To do this, use the netsh command. The name may be different depending on the OS language.

c:\> netsh advFirewall firewall show rule name="Remote Service Management (RPC)"

Rule Name:                            Remote Service Management (RPC)
----------------------------------------------------------------------
Enabled:                              No
Direction:                            In
Profiles:                             Domain,Private,Public
Grouping:                             Remote Service Management
LocalIP:                              Any
RemoteIP:                             Any
Protocol:                             TCP
LocalPort:                            RPC
RemotePort:                           Any
Edge traversal:                       No
Action:                               Allow
Ok.

This output means the rule is disabled, and the RPC is not available remotely. Note the profiles to which this rule applies. Make sure you use the correct firewall profile, especially if it is customized in an Active Directory domain environment. Enable this rule for RemoteIP *Any*, or your desired source subnet/ip, if your monitoring station is located in another network segment.

To enable it, see the following codeblock:

c:\> netsh advFirewall firewall set rule name="Remote Service Management (RPC)" new enable=yes

Updated 1 rule(s).
Ok.

Windows Server 2003 does not have predefined rules like Windows Server 2008. It defines a few services that contain the necessary settings. Make sure that the following ports are open, so they can be accessed from the source machine:

Port Service
445 MS RPC

The default configuration contains the file and print services that enables port 445. If you don't need the other services, disable them. Again, use the netsh command to enable this service:

c:\>netsh firewall set service type=FILEANDPRINT mode=ENABLE
Ok.

c:\>netsh firewall show portopening

Port configuration for Domain profile:
Port   Protocol  Mode     Name
-------------------------------------------------------------------
139    TCP       Enable   NetBIOS Session Service
445    TCP       Enable   SMB over TCP
137    UDP       Enable   NetBIOS Name Service
138    UDP       Enable   NetBIOS Datagram Service

Port configuration for Standard profile:
Port   Protocol  Mode     Name
-------------------------------------------------------------------
139    TCP       Enable   NetBIOS Session Service
445    TCP       Enable   SMB over TCP
137    UDP       Enable   NetBIOS Name Service
138    UDP       Enable   NetBIOS Datagram Service
3389   TCP       Enable   Remote Desktop

See the notes about dynamically assigned ports for DCOM RPC at the end of this page.

Remote accessibility of services

Test the availability of the necessary services on the monitored machine. Make sure to run these commands as the monitoring user:

Remote Registry

C:\>sc \\<server> query RemoteRegistry

SERVICE_NAME: RemoteRegistry
        TYPE               : 20  WIN32_SHARE_PROCESS
        STATE              : 4  RUNNING
                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

Remote Procedure Call (RPC)

C:\>sc \\<server> query RpcSs

SERVICE_NAME: RpcSs
        TYPE               : 20  WIN32_SHARE_PROCESS
        STATE              : 4  RUNNING
                                (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

DCOM

If the previous checks were successful, but slow (1s), verify that firewall profile allows DCOM (Port 135). Make sure the rule is valid for your source network and the current active firewall profile.

Check the current active profile for the Advanced Windows Firewall:

c:\>netsh advFirewall show currentprofile

Public Profile Settings:
----------------------------------------------------------------------
State                                 ON
Firewall Policy                       BlockInbound,AllowOutbound
LocalFirewallRules                    N/A (GPO-store only)
LocalConSecRules                      N/A (GPO-store only)
InboundUserNotification               Disable
RemoteManagement                      Disable
UnicastResponseToMulticast            Enable

Logging:
LogAllowedConnections                 Disable
LogDroppedConnections                 Disable
FileName                              %systemroot%\system32\LogFiles\Firewall\pfirewall.log
MaxFileSize                           4096

Ok.

The predefined Performance Logs and Alerts (DCOM-In) rule enables DCOM, depending on the profile subnet:

c:\>netsh advFirewall firewall show rule name="Performance Logs and Alerts (DCOM-In)"

Rule Name:                            Performance Logs and Alerts (DCOM-In)
----------------------------------------------------------------------
Enabled:                              No
Direction:                            In
Profiles:                             Domain
Grouping:                             Performance Logs and Alerts
LocalIP:                              Any
RemoteIP:                             Any
Protocol:                             TCP
LocalPort:                            135
RemotePort:                           Any
Edge traversal:                       No
Action:                               Allow

Rule Name:                            Performance Logs and Alerts (DCOM-In)
----------------------------------------------------------------------
Enabled:                              Yes
Direction:                            In
Profiles:                             Private,Public
Grouping:                             Performance Logs and Alerts
LocalIP:                              Any
RemoteIP:                             LocalSubnet
Protocol:                             TCP
LocalPort:                            135
RemotePort:                           Any
Edge traversal:                       No
Action:                               Allow
Ok.

Make sure the rule matches your setup, and enable or modify the rule. Another default rule enables DCOM for any subnet. Check your Policy before you use this.

c:\>netsh advFirewall firewall show rule name="DFS Management (DCOM-In)"

Rule Name:                            DFS Management (DCOM-In)
----------------------------------------------------------------------
Enabled:                              No
Direction:                            In
Profiles:                             Domain,Private,Public
Grouping:                             DFS Management
LocalIP:                              Any
RemoteIP:                             Any
Protocol:                             TCP
LocalPort:                            135
RemotePort:                           Any
Edge traversal:                       No
Action:                               Allow
Ok.

User authentication and permissions

To access the performance data of a machine remotely, the querying user must have access. For that purpose, Windows 2003/2008 provides the built-in group Performance Monitor Users. The monitoring user must be a member of this group and be known on both machines. On the source machine this user must be a member of the Performance Monitor Users for access rights to certain registry paths.

Scenario: source and target machines are members of a domain

If the source and target monitoring machines are members of the same Active Directory domain, follow these setup steps:

  1. Create a Domain Monitoring User (ID monitor) in Active Directory.
  2. Add this user to the local group Performance Monitor Users on each machine you query, and on the machine that executes the AppMon Collector. If this is a large number of machines, use the central group policy.
  3. Configure the AppMon Service to run as the Monitor user.

Scenario: source and target machines are not members of a domain

If the machines are single hosts with independent user accounts and security policies, follow these setup steps:

  1. Create the monitor user on all machines, with the same ID and password.
  2. Add the monitor user to the local group Performance Monitor Users on all machines.
  3. Configure the AppMon Service to run as the monitor user.

Scenario: source machine is not in a domain, target machine is

If the monitoring machine is not a member of a domain but the monitored machine is, you can't use a domain user to execute the performance metric queries. Use the same setup that you would for two independent machines.

Scenario: machines in different domains or forests

If the machines are in different domains, you need appropriate trusts and delegations established in the forest. This is similar to the first scenario. You may want to use global and domain local groups, instead of single users for granting rights.

Hardening the monitored systems

User permissions and groups

The monitor user needs no additional rights. In production systems, limit permissions to those who are necessary and adapt your security policy. By default, a newly created user belongs to the Users or Domain Users group, which isn't required for the monitoring tasks. The users should be removed from those groups. They should be members of the Performance Monitor Users.

File permissions

On the AppMon Collector machine, the monitor user running with the Collector service requires full access to the AppMon Collector installation directory. If you have installed the collector with another user, like an Administrative user, change the file permissions accordingly.

Local and global security policy

Use the monitor for monitoring only. Deny the interactive login locally and remotely for this user. The user must log in as a service. To do this, edit your applicable local or global (in an Active Directory domain) security policy to deny login rights to the Monitor user.

For single hosts, follow these steps:

  1. Edit the Local Computer Policy on each monitored machine using MMC.
  2. Go to Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment.
  3. Add the user or group to the Deny Login Local policy. If you are in a domain, do this centrally.

Testing performance monitor query

To test the setup, follow these steps and do not restart the AppMon Collector service. If the setup is successful, the AppMon Collector should query Windows performance data.

Execute the commands on the monitoring machine (the machine on which the AppMon collector is running).

Log in with the monitor user (recommended) or open a command shell as the monitor user. Use the *runas* command.

C:\>runas /user:monitor@yourdomain.local cmd
Enter the password for monitor@yourdomain.local:
Attempting to start cmd as user "monitor@yourdomain.local" ...

In the newly opened command shell use typeperf to test if you can query the remote machine performance counters, such as one memory counter:

C:\>typeperf -sc 2 "\\<monitored machine name>\Memory\Available MBytes"

"(PDH-CSV 4.0)","\\2008VMX64\Memory\Available MBytes"
"10/01/2010 15:22:09.602","2220.000000"
"10/01/2010 15:22:10.605","2220.000000"

The command completed successfully.

If you can successfully query the performance counters, you should be able to set up the AppMon Windows Performance Monitor. If you do not get any data, check your firewall and permission settings. Also, make sure you use the correct user to execute the query.

Use thetypeperf command to check which performance counters and objects are available on your monitored machines. To add custom counters that are not predefined in AppMon, use this command to identify them or export them to a text file, rather than using *perfmon.exe*. To query all counters for a specific object use the following code:

c:\> typeperf \-q Processor
\Processor\(*)\% Processor Time
\Processor\(*)\% User Time
\Processor\(*)\% Privileged Time
\Processor\(*)\Interrupts/sec
\Processor\(*)\% DPC Time
\Processor\(*)\% Interrupt Time
\Processor\(*)\DPCs Queued/sec
\Processor\(*)\DPC Rate
\Processor\(*)\% Idle Time
\Processor\(*)\% C1 Time
\Processor\(*)\% C2 Time
\Processor\(*)\% C3 Time
\Processor\(*)\C1 Transitions/sec
\Processor\(*)\C2 Transitions/sec
\Processor\(*)\C3 Transitions/sec

Configuring the AppMon Collector

Configure the AppMon Collector Service to run as the monitor user. The Windows plugin executes as this user from the Collector and derives its permissions. Make sure that the user that runs with the collector service has full access rights on the AppMon Collector install directory.

For more information, see Collector Configuration..