Log content auto-discovery v1

By default, Dynatrace auto-discovers all new log files. All log files must meet certain requirements to be auto-discovered.

Default auto-discovery

Dynatrace auto-discovers, analyzes, and stores logs every 60 seconds. You can modify this frequency in the Log Monitoring configuration file.

By default, log files are auto-discovered and analyzed if they are in:

• Windows System Log
• Windows Security Log
• Windows Application Log
• /var/log/messages
• /var/log/syslog

Auto-discovery requirements

A log file must meet all of the following requirements in order to be auto-discovered:

• The log file must be opened by an important process. See, Which are the most important processes?

• The logs must have the character encoding supported. By default, the supported encoding is UTF-8. Other supported types include UTF-8 BOM, as well as UTF-16LE and UTF-16BE if the files contain the bite-order mark (BOM).

  Binary logs and unsupported timestamp

  Binary log files and log files that contain an unsupported timestamp will be detected automatically but will not be analyzed and will not be stored (only the file status will be reported). In addition, files with unsupported timestamp will be automatically timestamped with the file reading time. See, Log Monitoring configuration file.

• The log file must be at least 0.5 KB in size.

• The log file must have been updated (written to) in the last 7 days.
  Log files that have not been updated in the past 7 days while the log monitoring is active will not be visible on dashboards.

• The log file must be in the actual log or logs folder or in it's sub folders:

  • Valid path examples:
    c:\log\log_file.txt
c:\logs\NewFolder\log_file.txt
  • Invalid path example:
    c:\log\NewFolder\NewFolder\log_file.txt

  or the log filename must contain a log string preceded or followed by the period (.) or underscore (_) character:

  • Valid filename examples:
    c:\NewFolder\abc.log
c:\NewFolder\0865842.log.txt
  • Invalid filename example:
    c:\NewFolder\logfile.txt

What might prevent logs to appear on the server?

• Detected over 200 rotated log file groups for a process.

  Dynatrace detects rotation scheme for log files and reports all log files under as a group under one name, which typically maps to many files on disk. Large number of rotated file groups typically means that Dynatrace did not recognize rotation pattern correctly and reports each physical file separately as a group. Once 200 reported rotated log file groups is reached, auto detection is turned off for this process. To resolve this issue, you can:

  • Define a custom file which properly captures variable parts of file name.
  • Push up the limit in agent configuration, FilesInGroup property (Log Monitoring configuration file).

• Fast growing files are skipped.

  In a scenario where a log file grows very quickly (at a pace over 10 MB/s), it's content might be skipped. The agent will continue to send the log file as long as both, the network and the server, can handle such load. Note that 10 MB/s with typical compression is approximately 10 Mbps of upload traffic.

• File name or path doesn't match typical log naming.

  The agent checks if logs match file name and path pattern that is typical for log files. If there is no match, the file will not be reported and sent to the server. This is needed to avoid false positives on detection of files as logs, and to prevent pulling non-log data from hosts. To remedy this you can set rules in the agent configuration, AutomaticFile property (Log Monitoring configuration file).

• There are symbolic links in the file or the paths

  This limitation applies to custom files in which the user points to a path that contains symbolic links. The physical path of the file pointed by symbolic the link must meet criteria for a log. Otherwise, the symbolic links could be used to read non-log data from host.

• File size is below 500 bytes.

Additional limitations for automatically detected files:

• File cannot be deleted earlier than a minute after creation.
• Files must be appended (old content is not uploaded).
• Files must have text content.
• Log files must be opened constantly (not just for short periods of adding log entry).
• Log Files must be opened in write mode.