Remote Unix Monitoring 2.0

Remote Unix Monitoring 2.0

Extension that remotely collects Unix OS data by executing commands via SSH.

Extension

Product information
Release notes

Overview

Overview

Unix/Linux is often used to run critical applications. A OneAgent is the best way to monitor such systems, however this is not always possible. Legacy unsupported versions or agreements with vendors may prevent you from installing a OneAgent on these important hosts. In these cases you can use the Remote Unix Monitoring extension to collect valuable telemetry on your hosts and the applications they are running. It works by connecting to your Unix/Linux hosts over SSH and running commands.

Use cases

Monitor and alert on important host metrics
Monitor the availability and resource usage of your key processes and applications
Identify host availability issues

Details

Metrics

Review the list of feature sets to see which metrics are collected. Note that for some distributions different metrics/metric keys may be used for some types of metrics due to similar but not identical metrics being available on that distribution as compared to what is more commonly collected.

Log events

Top process data

When configured, details about the 'top' processes running over time will be ingested as log events. These will include more complete data about the process (including the process ID) as compared to what the metrics alone will provide.

Filtered process PID change

When monitoring a group of filtered processes, if the set of PIDs in that group changes an event will be reported.

Connection issues

An availability event will be reported in cases of connection issues to the target host.

Compatibility information

The following OSs are explicitly supported and can be selected in the configuration:

AIX
Centos
FreeBSD
HP-UX
MacOS
Oracle Linux
Red Hat Enterprise Linux (RHEL)
Solaris
Suse

Many other distributions such as Ubuntu can be monitored by selecting 'Generic Linux.'

As a goal of this extension is to provide visibility into systems that cannot be monitored by a OneAgent it does not have the same support policy as other areas of Dynatrace where we support what the vendor supports. Rather we try to support as many distributions as possible regardless of their support status to the best of our ability.

Installation

Requirements

Minimum Dynatrace environment and ActiveGate version 1.269
An OS in the listed of the supported distributions
SSH enabled and reachable from assigned ActiveGates
A user with permission to connect and run the required commands
Either a valid password or key for authentication
- Note: Kerberos authentication (e.g. Centrify) is not supported for password authentication. In these cases using a certificate is recommended

Dynatrace configuration

Find 'Remote Unix Monitoring 2.0' in the in-product Extensions or Hub page and activate (if offline you can download the extension from this Hub page in the 'Versions' section and install as a custom extension).

Monitoring configurations

Once activated in your environment you can create monitoring configurations. Each monitoring configuration can have one or more remote unix target hosts configured.

First select the desired ActiveGate group that will run the monitoring configuration.

For each cluster configure a Remote Unix Endpoint:

Hostname: hostname or IP address to connect to
Port: port where the SSH server is listenening (default 22)
Host alias: an optional alias for the target host, defaults to hostname configured
Operating system
Authentication
- Password: supports credential vault
- Key contents: SSH private key contents as copy-pasted from key file
- Key path: a path to a key file. Must be present and readable on any ActiveGate to work
Additional properties: additional properties to be added as dimensions to ingested metrics
Top process count: when reporting top process metrics how many should be collected (default 10)
Report as log events: report more detailed top process data as log events
Process filters: a set of process filters to collect filtered process data for
- Group key: process group to which matches will be assigned
- Pattern: regex to match the process commands
- User: optional user filter
Mount filters: filteres to control which mounts to report metrics for
Advanced settings
- Custom PATH: if the default non-login path does not have all required commands, specify a custom PATH to be used when looking for commands
- SSH connection behavior: disconnect on each execution or reuse between runs and keep open
- RSA2 algorithm: some older systems require RSA2 to be disabled. See here for details.
- Top mode
  - Metrics that require top for process level metrics will be run in threads mode (-H) and report metrics at the process thread level
  - Can be used if in multi-processor environments percentages of CPU greater than 100% are reported
- Max concurrent channels: Determines number of simultaneous channels that can be open over SSH connection. Default and Max: 5.
  - Some systems may experience channels errors due to simultaeous channels being used. In these cases you can try lowering this value.
- Log command output: command output will be recorded in debug logs to assist with troubleshooting. DEBUG logging must be enabled.

Feature sets

Review the available feature sets to determine which you want to enable and collect. The enabled feature sets will effect which commands are run to avoid running commands that are not needed.

Troubleshooting

Required commands

Commands used are based on what is available in a 'vanilla' installation of the given OS, so typically no additional installs will be required.

Required commands must be available on the 'PATH' set for non-login shells, which can be different from login shells which is what you use when manually SSHing into a server. If you see log entries that commands are missing check that the command is available and that it is in the 'PATH' for non-login shells. If not, you can either correct this via configuration on the host or use the custom path configuration field to manually set a path to use.

You can test commands in non-login shells by running the ssh command with the command to run appended:

ssh <host> <command>

To view the path you can try:

ssh <host> "echo $PATH"

Debian Based Linux (Generic Linux)

vmstat
w
df
cat
top
pgrep
iostat
- for Red Hat Enterprise Linux systems you may need to install the sysstat package to collect disk IO metrics. No alternatives that exist by default were found.
ip or netstat
- older OSs don't have required stats in 'ip' command and will 'fall back' to netstat

Solaris

vmstat
w
df
netstat
ps
grep
prstat
kstat
mpstat
iostat
swap
prtconf

AIX

vmstat
w
df
netstat
ps
grep
svmon
mpstat
iostat

FreeBSD

vmstat
w
df
netstat
ps
top
grep
iostat

HP-UX

vmstat
w
df
netstat
ps
grep
swapinfo
top
sar
machinfo

Extension Logs

For some authentication errors you may need to check authentication/access logs on the target host for details as to why a connection is failing.

Debug logging should only be set when necessary and then reverted to avoid excessive logging.

Licensing

Licensing

There is no charge for obtaining the extension, only for the data (metrics & events) that the extension ingests. The details of license consumption will depend on which licensing model you are using. This will either be Dynatrace classic licensing or the Dynatrace Platform Subscription (DPS) model.

Metrics

License consumption is based on the number of metric data points ingested. The following formula will provide approximate annual data points ingested assuming all feature sets are enabled:

Calculation estimates yearly DDU usage:

(21 + (4 * <number of CPUs>) + (2 * <matching_filtered_processes>) + (2 * <process_groups>) + (3 * <matching_mounts>) + (8 * <network_interfaces>) + (2 * <disks>)) * 525.6

See description of 'Process filter' configuration for explanation of what a group and filtered processes will be
Matching mounts will be those that meet criteria of 'Mounts to include' and 'Mounts to exclude' configurations

Events

Each custom event reported will consume license. The number of reported custom events will be based on the frequency of things like connection errors where an availability event is reported.

Classic licensing

In the classic licensing model, custom events will consume Davis Data Units (DDUs) at the rate of .001 DDUs per metric data point.

Multiply estimated ingested log records by .001 to estimate DDU usage from log records.

Log records

Log records will be ingested for top process records if enabled based on the configured number of top process records once per minute.

Log management and analytics (powered by Grail)

License consumption is based on the size (in bytes) of data ingested & processed, retained, and queried so there is not a single formula to estimate the total consumption from this extension. Consult the log management and analytics documentation for details on the other dimensions that will effect license consumption.

Classic licensing

In the classic licensing model, log record ingestion will consume Davis Data Units (DDUs) at the rate of 100 DDUs per Gigabyte of log records ingested.

Log monitoring classic

In log monitoring classic, license consumption is based on the number of ingested log records.

Classic licensing

In the classic licensing model, log record ingestion will consume Davis Data Units (DDUs) at the rate of .0005 DDUs per ingested log record.

Multiply estimated ingested log records by .0005 to estimate DDU usage from log records.

Subscribe to new releases

Copy to clipboard

Extension content

Content typeNumber of items included

screen logs cards

1

screen properties

1

metric metadata

51

screen layout

2

generic relationship

1

dashboards

1

list screen layout

2

screen chart groups

3

screen metric tables

11

generic type

2

screen events cards

1

screen entities lists

3

Feature sets

Below is a complete list of the feature sets provided in this version. To ensure a good fit for your needs, individual feature sets can be activated and deactivated by your administrator during configuration.

Feature setsNumber of metrics included

Metric name	Metric key	Description	Unit
Mount used	remote_unix.mount_used	Mount space in use	Byte
Mount capacity	remote_unix.mount_capacity	Overall capacity of the mount	Percent
Mount available	remote_unix.mount_available	Mount space available	Byte

Metric name	Metric key	Description	Unit
CPU Utilization	remote_unix.cpu_utilization	Overall CPU utilization	Percent
CPU User	remote_unix.cpu_user	Percentage of time the processor spent executing user/application code	Percent
CPU System	remote_unix.cpu_system	Percentage of time the processor spent executing system/kernel	Percent
CPU Idle	remote_unix.cpu_idle	Percentage of time the processor was idle	Percent
Individual CPU user time	remote_unix.individual_cpu_time_user	Percentage of time individual CPU spent executing user/application code	Percent
Individual CPU system time	remote_unix.individual_cpu_time_system	Percentage of time individual CPU spent executing system/kernel code	Percent
Individual CPU idle time	remote_unix.individual_cpu_time_idle	Percentage of time individual CPU was idle	Percent
Individual CPU iowait time	remote_unix.individual_cpu_time_iowait	Percentage of time individual CPU spent waiting on IO operations	Percent

Metric name	Metric key	Description	Unit
Zpool size	remote_unix.zpool_size	Zpool total size	Byte
Zpool allocated	remote_unix.zpool_allocated	Zpool space allocated	Byte
Zpool free	remote_unix.zpool_free	Zpool space free	Byte
Zpool capacity	remote_unix.zpool_capacity	Zpool used percent	Percent

Metric name	Metric key	Description	Unit
Disk latency (average)	remote_unix.disk_average_latency	Disk latency (average)	MilliSecond

Metric name	Metric key	Description	Unit
Network bytes receieved	remote_unix.network_bytes_received_count	Network traffic (in bytes) incoming per interface	Byte
Network packets received	remote_unix.packets_received_count	Network traffic (in Packets) incoming per interface	Count
Network errors incoming	remote_unix.network_errors_incoming_count	Network errors on incoming traffic per interface	Count
Packets dropped incoming	remote_unix.packets_dropped_incoming_count	Incoming packets dropped per interface	Count
Network bytes sent	remote_unix.network_bytes_sent_count	Outbound network traffic (in bytes) per interface	Byte
Network packets sent	remote_unix.packets_sent_count	Outbound network traffic (in packets) per interface	Count
Network errors outgoing	remote_unix.network_errors_outgoing_count	Network errors on outgoing traffic per interface	Count
Packets dropped outgoing	remote_unix.packets_dropped_outgoing_count	Outgoing packets dropped per interface	Count

Metric name	Metric key	Description	Unit
Availability	remote_unix.availability	Availability of host based on initial SSH connection	Percent
Waiting processes	remote_unix.waiting_processes	Number of processes waiting to for processor time	Count
User count	remote_unix.users	Number of users on the system	Count
Load average (1 min)	remote_unix.load_avg_1_min	One minute load average	Count
Load average (5 min)	remote_unix.load_avg_5_min	Five minute load average	Count
Load average (15 min)	remote_unix.load_avg_15_min	15 minute load average	Count

Metric name	Metric key	Description	Unit
Top process CPU usage	remote_unix.top_process_cpu	CPU usage of a 'top' resource consuming detected process	Percent
Top process memory usage	remote_unix.top_process_size	Memory usage of a 'top' resource consuming detected process	Byte

Metric name	Metric key	Description	Unit
Paged in	remote_unix.paged_in	Memory pages swapped in from disk	PerSecond
Paged out	remote_unix.paged_out	Memory pages swapped to disk	PerSecond
Physical memory free	remote_unix.physical_memory_free	Amount of memory free as seen by OS	Byte
Physical memory used percent	remote_unix.physical_memory_used_percent	Percentage of memory used as seen by OS	Percent
Swap free	remote_unix.swap_free	Amount of swap space free	Byte
Swap total	remote_unix.swap_total	Total swap space available on system	Byte
Swap used percent	remote_unix.swap_used_percent	Percentage of available swap space in use	Percent
Swap used percent	remote_unix.swap_free_percent	Percentage of available swap space not in use	Percent

Metric name	Metric key	Description	Unit
Disk read	remote_unix.disk_read_count	Amount of data read from disk	Byte
Disk write	remote_unix.disk_write_count	Amount of data written to disk	Byte
Disk read operations	remote_unix.disk_read_ops	Reads from disk	PerSecond
Disk write operations	remote_unix.disk_write_ops	Writes to disk	PerSecond
Bytes per transfer	remote_unix.bytes_per_transfer	Bytes per transfer	Byte
Disk transfers	remote_unix.transfers	Disk transfers	Count

Full version history

To have more information on how to install the downloaded package, please follow the instructions on this page.

ReleaseDate

Full version history

considers buffer and cache space in reporting available memory on some older Unix distributions
handles reporting of values with the largest units (E, Z, and Y)
fix bug in reporting of MacOS disk transfers metric
adds missing metric metadata for some MacOS-specific metrics

Full version history

Corrects issue in build process that would prevent the previous 2.2.7 version from running on old ActiveGate OSs (e.g. RHEL 7.9)

Full version history

Sets source entity metadata for all metrics to work with management zones

Full version history

Default the optional remote host group value to "default"
- Primarily to ensure metric tables/charts/selectors will not break when not set

Full version history

Fixes bug in AIX where certain output of the 'df' command could cause processing of mount data to break
Increases filtered process pattern length limit to 1000

Full version history

New Solaris Z Pools feature set
Scalability-focused features
- Monitoring configuration-wide 'default' authentication possible
- Configuration available for a 'remote host group' entity to help with organizing remote unix entities
- Raises endpoint limit to 500
- Must use the new Task bucket size configuration to break the endpoints into smaller assignable task units
Fix for issue on AIX where memory metric absolute values were incorrectly scaled to be larger than their actual values (did not affect 'percentage' based metrics)

Full version history

Fix issue for some distributions that would result in missing data for the disk-level "bytes written" metric

Full version history

New 'extended-disk' feature set
- Supported for most 'standard' Debian based Linux distributions
- Includes an additional run of iostat with the 'extended' option to report latency
Attempts to include IP addresses and a detected hostname property
- Restore behavior present in original 1.0 extension

Full version history

Fix issue where configured key path may not have properly been used for authentication

Full version history

Fix bug with how failures in the fastcheck would be reported in some scenarios

Full version history

Initial public release