Unix/Linux is often used to run critical applications. A OneAgent is the best way to monitor such systems, however this is not always possible. Legacy unsupported versions or agreements with vendors may prevent you from installing a OneAgent on these important hosts. In these cases you can use the Remote Unix Monitoring extension to collect valuable telemetry on your hosts and the applications they are running. It works by connecting to your Unix/Linux hosts over SSH and running commands.
The following OSs are explicitly supported and can be selected in the configuration:
- AIX
- Centos
- FreeBSD
- HP-UX
- MacOS
- Oracle Linux
- Red Hat Enterprise Linux (RHEL)
- Solaris
- Suse
Many other distributions such as Ubuntu can be monitored by selecting 'Generic Linux.'
Metrics
- Host availability
- CPU utilization
- CPU utilization (User)
- CPU utilization (System)
- CPU utilization (Idle)
- CPU breakdown (User)
- CPU breakdown (System)
- CPU breakdown (Idle)
- CPU breakdown (IOwait)
- Waiting processes
- Paged in (memory)
- Paged out (memory)
- Memory free
- Memory used (percent)
- Swap free
- Swap total
- Swap used (percent)
- Swap free (percent)
- Top process cpu
- Top process size
- Filtered process cpu
- Filtered process size
- Filtered process match count
- Filtered process PIDs changed
- User count
- Mount used
- Mount capacity
- Mount available
- Network bytes
- Network packets
- Network errors
- Network packets dropped
- Uptime
- Disk reads
- Disk writes
- Load average (1 min)
- Load average (5 min)
- Load average (15min)
Approximate DDU Usage
Calculation estimates yearly DDU usage:
(21 + (4 * <number of CPUs>) + (2 * <matching_filtered_processes>) + (2 * <process_groups>) + (3 * <matching_mounts>) + (8 * <network_interfaces>) + (2 * <disks>)) * 525.6
- See description of 'Process filter' configuration for explanation of what a group and filtered processes will be
- Matching mounts will be those that meet criteria of 'Mounts to include' and 'Mounts to exclude' configurations
UI
Once configured and 'OK' states reported you can see the custom devices by going to the 'Technologies and processes' page and selecting the group for 'Remote Linux.' You'll find your instances under the groups you have configured.
Alerts
To create alerts go to settings and define custom events for alerting using the metrics asociated with this extension.
Requirements
- OS listed in 'Supported Distributions'
- SSH connection available on the host
- Password or key based authentication supported
- Credentials that permit running of commands to collect metrics
- None of the commands run should require superuser permissions
- Can run on Linux or Windows ActiveGates
- Target hosts that accept SSH connections with properly provisioned account
- Environment ActiveGate server - Extract the extension zip file to the plugin_deployment folder of the remote plugin module
- In a default installation this is done with
unzip -o -d /opt/dynatrace/remotepluginmodule/plugin_deployment custom.remote.python.remote_agent.zip
- Windows path:
C:\Program Files\dynatrace\remotepluginmodule\plugin_deployment
- Adjust the path if the Activegate was installed somewhere other than the default location
- Especially on Windows make sure that when unzipping the extension you do not 'double layer' the folders. It should be
.../plugin_deployment/custom.remote.python.remote_agent/<plugin source and files> NOT .../plugin_deployment/custom.remote.python.remote_agent/custom.remote.python.remote_agent/<plugin source and files>
- Browser - Upload the same extension zip file to your tenant
Settings > Monitored technologies > Custom extensions > Upload extension
Configuration
Create an endpoint for each cluster you would like to monitor. This is done in Settings > Monitored technologies > Custom extensions > Remote Linux Monitor.
The parameters are:
- Hostname
- e.g. IP address or hostname
- Username
- (Optional) Password
- For authenticating with username/password
- (Optional) Private key file path
- For authenticating with private key located on ActiveGate
- (Optional) Private key contents
- Contents of private key exactly as present in key file (i.e. with newlines and padding)
- This takes priority over the key path configuration
- Must be an RSA key
- (Optional) Private key file passphrase
- For authenticating with private key
- Persist SSH connection
- Create and end SSH connection during each execution or attempt to maintain and reuse connection
- (Optional) Disable RSA2 Algorithm
- Very old systems may require this setting if they do not support RSA2 or the server-sig-algs protocol extension
- OS Type
- Type of OS connecting to (determines which commands are run)
- Group
- Which group of custom devices to belong to
- Used in custom device ID creation
- (Optional) Device name
- Custom device name (defaults to value used for hostname)
- Used in custom device ID creation
- (Optional) Process filter
- Which specific processes to report metrics for
- <pattern>;<process_group>
- One entry per line (multiple allowed)
- CPU and Memory will be reported for processes that match the patterns, additionally you will get group level metrics for the number of processes matching that pattern (can be used for availability alerting) and when the set of PIDs matching that pattern have changed between extension runs
- Threads mode (top)
- Metrics that require top for process level metrics will be run in threads mode (-H) and report metrics at the process thread level
- Can be used if in multi-processor environments percentages of CPU greater than 100% are reported
- (Optional) Mounts to include
- Mounts patterns to capture metrics for (all included by default)
- (Optional) Mounts to exclude
- Mounts patterns to not capture metrics for (applied last i.e. overrides inclusion list)
- Additional properties
- Additional properties that will be reported on the custom device
- <key>=<value>
- One entry per line (multiple allowed)
- Fail on initial error
- When checked, if there is an error (e.g. authentication error on the first run) a configuration error will be reported in the UI (later errors handled and reported as availability metric)
- When unchecked all errors will be handled and reported as availability metric even on first run
- (Advanced) Custom PATH
- A path to be used when searching for the required commands the extension runs.
- The extension uses non-login shells meaning the path it uses can be different from what you would see when logging into the server. If you see missing data and command not found errors in the logs you may need to modify the server configuration to set the path for non-login shells or instead add the full path setting to this config.
- e.g.
/opt/csw/bin:/usr/xpg4/bin:/usr/sbin:/usr/bin:/usr/ucb
- (Advanced) Max channel threads
- Determines number of simultaneous channels that can be open over SSH connection. Default and Max: 5.
- (Advanced) Disable iostat command
- Disables collection of disk metrics through iostat (used when output is too large and cannot complete within acceptable time).
- Log level
- Set to DEBUG if you are troubleshooting issues, otherwise keep on INFO
Custom entities
Metrics will be reported to custom devices under the "Remote Linux" technology category. The Group and Device names will depend on your configuration.
Required commands
Commands used are based on what is available in a 'vanilla' installation of the given OS, so typically no additional installs will be required.
Required commands must be available on the 'PATH' set for non-login shells, which can be different from login shells which is what you use when manually SSHing into a server. If you see log entries that commands are missing check that the command is available and that it is in the 'PATH' for non-login shells. If not, you can either correct this via configuration on the host or use the custom path configuration field to manually set a path to use.
You can test commands in non-login shells by running the ssh command with the command to run appended:
ssh <host> <command>
To view the path you can try:
ssh <host> "echo $PATH"
Debian Based Linux (Generic Linux)
- vmstat
- w
- df
- cat
- top
- pgrep
- iostat
- for Red Hat Enterprise Linux systems you may need to install the sysstat package to collect disk IO metrics. No alternatives that exist by default were found.
- ip or netstat
- older OSs don't have required stats in 'ip' command and will 'fall back' to netstat
Solaris
- vmstat
- w
- df
- netstat
- ps
- grep
- prstat
- kstat
- mpstat
- iostat
- swap
- prtconf
AIX
- vmstat
- w
- df
- netstat
- ps
- grep
- svmon
- mpstat
- iostat
FreeBSD
- vmstat
- w
- df
- netstat
- ps
- top
- grep
- iostat
HP-UX
- vmstat
- w
- df
- netstat
- ps
- grep
- swapinfo
- top
- sar
- machinfo
Extension Logs
The logs under %PROGRAMDATA% (windows) or /var/lib (linux) give us more details in case of failures
The full path is /var/lib/dynatrace/remotepluginmodule/log/remoteplugin/custom.remote.python.remote_agent/RemoteAgentExtension.log
or
C:\ProgramData\dynatrace\remotepluginmodule\log\remoteplugin\custom.remote.python.remote_agent/RemoteAgentExtension.log
For some authentication errors you may need to check authentication/access logs on the target host for details as to why a connection is failing.
Debug logging should only be set when necessary and then reverted to avoid excessive logging.
