Remote Unix Monitoring 2.0

Overview

Unix/Linux is often used to run critical applications. A OneAgent is the best way to monitor such systems, however this is not always possible. Legacy unsupported versions or agreements with vendors may prevent you from installing a OneAgent on these important hosts. In these cases you can use the Remote Unix Monitoring extension to collect valuable telemetry on your hosts and the applications they are running. It works by connecting to your Unix/Linux hosts over SSH and running commands.

• Monitor and alert on important host metrics
• Monitor the availability and resource usage of your key processes and applications
• Identify host availability issues

Metrics

Review the list of feature sets to see which metrics are collected. Note that for some distributions different metrics/metric keys may be used for some types of metrics due to similar but not identical metrics being available on that distribution as compared to what is more commonly collected.

Log events

Top process data

When configured, details about the 'top' processes running over time will be ingested as log events. These will include more complete data about the process (including the process ID) as compared to what the metrics alone will provide.

Filtered process PID change

When monitoring a group of filtered processes, if the set of PIDs in that group changes an event will be reported.

Connection issues

An availability event will be reported in cases of connection issues to the target host.

The following OSs are explicitly supported and can be selected in the configuration:

• AIX
• Centos
• FreeBSD
• HP-UX
• MacOS
• Oracle Linux
• Red Hat Enterprise Linux (RHEL)
• Solaris
• Suse

Many other distributions such as Ubuntu can be monitored by selecting 'Generic Linux.'

As a goal of this extension is to provide visibility into systems that cannot be monitored by a OneAgent it does not have the same support policy as other areas of Dynatrace where we support what the vendor supports. Rather we try to support as many distributions as possible regardless of their support status to the best of our ability.

Requirements

• Minimum Dynatrace environment and ActiveGate version 1.269
• An OS in the listed of the supported distributions
• SSH enabled and reachable from assigned ActiveGates
• A user with permission to connect and run the required commands
• Either a valid password or key for authentication
  • Note: Kerberos authentication (e.g. Centrify) is not supported for password authentication. In these cases using a certificate is recommended

Dynatrace configuration

Find 'Remote Unix Monitoring 2.0' in the in-product Extensions or Hub page and activate (if offline you can download the extension from this Hub page in the 'Versions' section and install as a custom extension).

Monitoring configurations

Once activated in your environment you can create monitoring configurations. Each monitoring configuration can have one or more remote unix target hosts configured.

First select the desired ActiveGate group that will run the monitoring configuration.

For each cluster configure a Remote Unix Endpoint:

• Hostname: hostname or IP address to connect to
• Port: port where the SSH server is listenening (default 22)
• Host alias: an optional alias for the target host, defaults to hostname configured
• Operating system
• Authentication
  • Password: supports credential vault
  • Key contents: SSH private key contents as copy-pasted from key file
  • Key path: a path to a key file. Must be present and readable on any ActiveGate to work
• Additional properties: additional properties to be added as dimensions to ingested metrics
• Top process count: when reporting top process metrics how many should be collected (default 10)
• Report as log events: report more detailed top process data as log events
• Process filters: a set of process filters to collect filtered process data for
  • Group key: process group to which matches will be assigned
  • Pattern: regex to match the process commands
  • User: optional user filter
• Mount filters: filteres to control which mounts to report metrics for
• Advanced settings
  • Custom PATH: if the default non-login path does not have all required commands, specify a custom PATH to be used when looking for commands
  • SSH connection behavior: disconnect on each execution or reuse between runs and keep open
  • RSA2 algorithm: some older systems require RSA2 to be disabled. See here for details.
  • Top mode
    • Metrics that require top for process level metrics will be run in threads mode (-H) and report metrics at the process thread level
    • Can be used if in multi-processor environments percentages of CPU greater than 100% are reported
  • Max concurrent channels: Determines number of simultaneous channels that can be open over SSH connection. Default and Max: 5.
    • Some systems may experience channels errors due to simultaeous channels being used. In these cases you can try lowering this value.
  • Log command output: command output will be recorded in debug logs to assist with troubleshooting. DEBUG logging must be enabled.

Feature sets

Review the available feature sets to determine which you want to enable and collect. The enabled feature sets will effect which commands are run to avoid running commands that are not needed.

Required commands

Commands used are based on what is available in a 'vanilla' installation of the given OS, so typically no additional installs will be required.

Required commands must be available on the 'PATH' set for non-login shells, which can be different from login shells which is what you use when manually SSHing into a server. If you see log entries that commands are missing check that the command is available and that it is in the 'PATH' for non-login shells. If not, you can either correct this via configuration on the host or use the custom path configuration field to manually set a path to use.

You can test commands in non-login shells by running the ssh command with the command to run appended:

ssh <host> <command>

To view the path you can try:

ssh <host> "echo $PATH"

Debian Based Linux (Generic Linux)

• vmstat
• w
• df
• cat
• top
• pgrep
• iostat
  • for Red Hat Enterprise Linux systems you may need to install the sysstat package to collect disk IO metrics. No alternatives that exist by default were found.
• ip or netstat
  • older OSs don't have required stats in 'ip' command and will 'fall back' to netstat

Solaris

• vmstat
• w
• df
• netstat
• ps
• grep
• prstat
• kstat
• mpstat
• iostat
• swap
• prtconf

AIX

• vmstat
• w
• df
• netstat
• ps
• grep
• svmon
• mpstat
• iostat

FreeBSD

• vmstat
• w
• df
• netstat
• ps
• top
• grep
• iostat

HP-UX

• vmstat
• w
• df
• netstat
• ps
• grep
• swapinfo
• top
• sar
• machinfo

Extension Logs

For some authentication errors you may need to check authentication/access logs on the target host for details as to why a connection is failing.

Debug logging should only be set when necessary and then reverted to avoid excessive logging.

Licensing

There is no charge for obtaining the extension, only for the data (metrics & events) that the extension ingests. The details of license consumption will depend on which licensing model you are using. This will either be Dynatrace classic licensing or the Dynatrace Platform Subscription (DPS) model.

Metrics

License consumption is based on the number of metric data points ingested. The following formula will provide approximate annual data points ingested assuming all feature sets are enabled:

Calculation estimates yearly DDU usage:

(21 + (4 * <number of CPUs>) + (2 * <matching_filtered_processes>) + (2 * <process_groups>) + (3 * <matching_mounts>) + (8 * <network_interfaces>) + (2 * <disks>)) * 525.6

• See description of 'Process filter' configuration for explanation of what a group and filtered processes will be
• Matching mounts will be those that meet criteria of 'Mounts to include' and 'Mounts to exclude' configurations

Events

Each custom event reported will consume license. The number of reported custom events will be based on the frequency of things like connection errors where an availability event is reported.

Classic licensing

In the classic licensing model, custom events will consume Davis Data Units (DDUs) at the rate of .001 DDUs per metric data point.

Multiply estimated ingested log records by .001 to estimate DDU usage from log records.

Log records

Log records will be ingested for top process records if enabled based on the configured number of top process records once per minute.

Log management and analytics (powered by Grail)

License consumption is based on the size (in bytes) of data ingested & processed, retained, and queried so there is not a single formula to estimate the total consumption from this extension. Consult the log management and analytics documentation for details on the other dimensions that will effect license consumption.

Classic licensing

In the classic licensing model, log record ingestion will consume Davis Data Units (DDUs) at the rate of 100 DDUs per Gigabyte of log records ingested.

Log monitoring classic

In log monitoring classic, license consumption is based on the number of ingested log records.

Classic licensing

In the classic licensing model, log record ingestion will consume Davis Data Units (DDUs) at the rate of .0005 DDUs per ingested log record.

Multiply estimated ingested log records by .0005 to estimate DDU usage from log records.