Unix system monitoring

Overview

The AppMon Unix Monitor monitors local or remote Unix systems using agentless system monitoring. The advantage of agentless monitoring is that you do not need to spend time and effort in deploying and maintaining Agents on every monitored system.

The Unix Monitor collects relevant monitoring data from the systems of interest through SSH, Rexec, or locally. The Monitor can be executed on AppMon Collectors running on Windows or Linux. The Windows Collector does not support the Local option.

The Monitor queries common Unix status commands and returns relevant information filtered from the resulting output stream. If every necessary command is activated on the monitored systems, the Monitor delivers values for every Measure it is configured to deliver. The operating system is automatically identified by executing the uname command on the monitored machine.

The Monitor is supported on Linux, Solaris, AIX, and HP-UX platforms.

The Unix Monitor Command List file contains executed commands, expected output, and descriptions of how the values are parsed.

Setup and Basic Configuration

To add a Unix Monitor to the System Profile:

  1. In the System Profile Preferences, select Monitors.
  2. Click Create to open the Monitor Type Chooser.
  3. Select Unix Monitor and click OK to open the Scheduled Monitor Editor.
  4. Specify a name and an optional description for the Monitor
  5. In the Settings area, configure the settings as described below.
  6. Add hosts as needed. See below for details.

Monitor settings

  • Method: The connection method can either be SSH or Local. See below more information on the SSH connection method. Use Local only if the target host is the same as the Collector where the Monitor is executed. If you select Local, all other settings except Customize Measurements (see below) become unavailable.

  • Authentication Method; For the SSH connection method. select either Password or PublicKey. For Password authentication, define these settings:

    • Port: The port through which the Monitor connects to the target host.
    • Username: The username to log on to the remote host.
    • Password: The password for the specified username.

    For PublicKey authentication, define these settings:

    • Port: The port for the SSH or RExec connection.
    • Username: The username to log on to the remote host.
    • Passphrase: The optional passphrase for the public key-file.
    • Key-File: The private key-file on the AppMon Collector.
  • Customize Measurements: See below.

Select host(s) that should be the target(s) of the monitor

A Monitor can be configured for multiple target hosts. Only one configuration is necessary; it is executed for all configured machines. This is especially useful in cluster environments that feature many similarly configured hosts.

Add Single hosts, hostgroups, or sites by clicking the + icon. Create a new host if the desired host is not shown. The Monitor runs for all contained hosts if you add a hostgroup or site.

Optionally, add an intersection of hostgroups or sites. The Monitor runs for all hosts that belong to all selected groups or sites. If the configuration would resolve a single host more than once (if Host1 is also contained in Hostgroup1), the Monitor is only executed once per host.

You can also add target hosts using the Server REST interface. The required POST request is located in the Tasks and Monitors group. The request documentation is based on the Swagger specification and embedded to the request form.

See Infrastructure for more information on how to manage hosts and hostgroups.

Examples

Configured Hosts As Shown in Hosts Table Monitor
Host1 Host1
Host1 Host2 Host1 and Host 1
Hostgroup 1 Host2 All hosts from Hostgroup 1 and also Host 2
Hostgroup 1 Hostgroup 2 All hosts from Hostgroup 1 and also Hostgroup 2
Hosts belonging to Hostgroup 1, Hostgroup 2 Host 1 All hosts that are both in Hostgroup 1 and Hostgroup 2 (an intersection of Hostgroup 1 and Hostgroup 2), plus Host 1

Customizing the execution schedule

Open the Schedule tab, then choose a predefined schedule or create your own schedule if none fits. Any schedules you create are available for the whole System Profile.

Centrally deploying a monitor to different global locations

Open the Schedule tab, then select the desired AppMon Collector or AppMon Server from the Execute on list. Only currently connected AppMon Collectors are shown in this list. The selection appears as Embedded Dynatrace Collector.

Note

If an AppMon Collector that is connected through a slow network or a Monitor that contains large library files is run on a remote machine, it is possible, though rare, that the first execution fails with a Missing Binaries error. This error should disappear on the next scheduled run of the Monitor.

Customized measurements

Use the Customize Measurements to customize the Measurements to a single CPU, NIC, or disk partition. When you select the check box in the Value column, three settings appear:

  • The CPU setting can be set to a CPU ID starting with 0. This requires the mpstat command on Solaris and the sar command on the other operating systems.
  • The NIC setting restricts the network measures to a single interface. Enter the value as shown in the netstat command; for example, eth0.
  • The Partition option restricts the disk measures to a specified partition. Enter the value as shown in the first column of the df command.

SSH connection with password authentication

For password authentication between the Monitor and the SSH server, the SSH server has to be configured to allow password authentication (not to be confused with keyboard-interactive authentication, which SSH also supports but which is not used by the Monitor). The following line must be part of the /etc/ssh/sshd_config file of the target system.

PasswordAuthentication yes

Password authentication fails if this is not set.

SSH connection with public key authentication

If you select PublicKey authentication, the public key must be added to the authorized-keys2 file on the SSH server, in the .ssh folder. The path to the key-file must be correct.

Here is an example how to generate an SSH key pair, if not already available, using OpenSSh:

Execute "ssh-keygen -t dsa"

or

Execute "ssh-keygen -t rsa"

Copy the id_dsa (or id_rsa) file to a location available to the AppMon Collector.  Specify the file path in the  Key-File field in the Monitor tab.

To add the public key to the authorized-keys2 file on the server, execute:

scp C:/.ssh/id_dsa.pub user@server:~/.ssh/authorized_keys2

Depending on the SSH-server setup, it might be necessary to use authorized_keys as the target file instead of authorized_keys2.

Only OpenSSH-style public keys can be used. Using keys saved by puttygen using Save private key (generating a .ppk file) results in an authentication error. To create OpenSSH keys with puttygen, select Export OpenSSH key from the puttygen Conversions menu.

Executed commands

The attached .xls file contains a list of the commands that are executed on the monitored systems. Further information can be found in the man pages.

After the activation of commands from the list, the Monitor should deliver values for each subscribed measure, with exceptions as described below. To collect a subset of the available measures, modify the selections in the Measures tab of the Scheduled Monitor Editor. You can also create incident rules if that option is available for the selected Measure.

Measures

The following measure can be subscribed.

  • Unix Memory Usage
    • Memory Available: Percentage of total memory that is usable (includes caches and buffers).
    • Memory Usage: Percentage of total memory that is being used.
    • Swap Usage: Percentage of total swap that is in use.
  • Unix Memory
    • Total Memory: Available total memory.
    • Used Memory: Currently used memory.
    • Free Memory: Currently free memory.
    • Total Swap: Available total swap space.
    • Used Swap: Currently used swap space.
    • Free Swap: Currently free swap space.
    • Pages Input: Paged/swapped in.
    • Pages Ouput: Paged/swapped out.
  • Unix CPU Usage
    • Processor Time: Total CPU usage (sum of user and system time).
    • User Time: Time spent running non-kernel code.
    • System Time: Time spent running kernel code.
    • Idle Time: Time idle.
    • I/O Wait Time: Time CPU was blocked while waiting for I/O operations.
  • Unix Process
    • Blocked: Number of processes in non-interruptible sleep.
    • Run Queue, current: Number of processes waiting for runtime.
    • Run Queue, last minute: Run Queue Length for the last minute.
    • Run Queue, last 5 minutes: Run Queue Length for the last 5 minutes.
    • Run Queue, last 15 minutes: Run Queue Length for the last 15 minutes.
    • Process Count: Number of processes currently shown in PS.
  • Unix Filesystem Usage
    • Temp Usage: Percentage of used space for file system mounted on /tmp.
    • Var Usage: Percentage of used space for file system mounted on /var.
    • Root Usage: Percentage of used space for file system mounted on /.
  • Unix Filesystem
    • Free Diskspace: Free disk space.
    • Total Diskspace: Total disk space.
    • Used Diskspace: Used disk space.
    • Disk Reads/sec: Reads per second.
    • Disk Writes/sec: Writes per second.
  • Unix Network
    • Bytes Total/sec: Total number of bytes received and transmitted.
    • Transmitted Bytes/sec: Total number of bytes transmitted.
    • Received Bytes/sec: Total number of bytes received.
    • Received Packets Total: Total number of packets received.
    • Received Packets OK: Number of successfully received packets.
    • Received Packets Error: Number of packets received with errors.
    • Received Packets Dropped: Number of dropped packets.
    • Transmitted Packets OK: Number of successfully transmitted packets.
    • Transmitted Packets Error: Number of packets received with errors.
    • Transmitted Packets Dropped: Number of dropped packets.
    • Transmitted Packets Total: Total number of packets transmitted.
    • Collisions: Collisions

Because of incompatibilities of some commands across different Unix systems, the following measures do not deliver values for the specified Unix system.

Linux:

  • Collisions (in Network Group)

Solaris:

  • Transmitted Packets Dropped (in Network Group)
  • Received Packets Dropped (in Network Group)
  • I/O Wait Time (in CPU Group)

AIX:

  • Bytes Total/sec (in Network Group)
  • Received Bytes/sec (in Network Group)
  • Transmitted Bytes/sec (in Network Group)
  • Transmitted Packets Dropped (in Network Group)
  • Received Packets Dropped (in Network Group)

HP-UX:

  • Total Swap (in Memory Group)
  • Free Swap (in Memory Group)
  • Used Swap (in Memory Group)
  • Root Usage (in Filesystem Group)
  • Temp Usage (in Filesystem Group)
  • Var Usage (in Filesystem Group)
  • Bytes Total/sec (in Network Group)
  • Received Bytes/sec (in Network Group)
  • Transmitted Bytes/sec (in Network Group)
  • Transmitted Packets Dropped (in Network Group)
  • Received Packets Dropped (in Network Group)
  • Transmitted Packets Error (in Network Group)
  • Received Packets Error (in Network Group)
  • Transmitted Packets OK (in Network Group)
  • Received Packets OK (in Network Group)
  • Collisions (in Network Group)
  • I/O Wait Time (in CPU Group)

Tested systems

The current AppMon release was tested to monitor the following operating systems:

  • AIX 5.2
  • HP-UX B.11.23
  • Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
  • SunOS 5.10
  • Ubuntu multiple (Ubuntu 12.04.5 LTS, Ubuntu 14.04.5 LTS, and others)

The Unix Monitor does not work in combination with CSH.