• Home
  • How to use Dynatrace
  • Log Monitoring v1
  • Log content auto-discovery v1

Log content auto-discovery v1

Legacy Log Monitoring v1

You are viewing documentation for Log Monitoring v1. Dynatrace Log Monitoring v1, is considered a legacy solution.

We strongly encourage you to switch to the latest Dynatrace Log Monitoring version.

  • Compare legacy Log Monitoring v1 to the latest version of Dynatrace log monitoring.
  • Switch to the latest version of Dynatrace log monitoring.

By default, Dynatrace auto-discovers all new log files. All log files must meet certain requirements to be auto-discovered.

Default auto-discovery

Dynatrace auto-discovers, analyzes, and stores logs every 60 seconds. You can modify this frequency in the Log Monitoring configuration file.

By default, log files are auto-discovered and analyzed if they are in:

  • Windows System Log
  • Windows Security Log
  • Windows Application Log
  • /var/log/messages
  • /var/log/syslog

Auto-discovery requirements

A log file must meet all of the following requirements in order to be auto-discovered:

  • The log file must be opened by an important process. See, Which are the most important processes?

  • The logs must have the character encoding supported. By default, the supported encoding is UTF-8. Other supported types include UTF-8 BOM, as well as UTF-16LE and UTF-16BE if the files contain the bite-order mark (BOM).

    Binary logs and unsupported time stamp

    Binary log files and log files that contain an unsupported time stamp will be detected automatically but will not be analyzed and will not be stored (only the file status will be reported). In addition, files with unsupported time stamp will be automatically timestamped with the file reading time. See, Log Monitoring configuration file.

  • The log file must be at least 0.5 KB in size.

  • The log file must have been updated (written to) in the last 7 days.
    Log files that have not been updated in the past 7 days while the log monitoring is active will not be visible on dashboards.

  • The log file must be in the actual log or logs folder or in it's sub folders:

    • Valid path examples:
      c:\log\log_file.txt
      c:\logs\NewFolder\log_file.txt
    • Invalid path example:
      c:\log\NewFolder\NewFolder\log_file.txt

    or the log filename must contain a log string preceded or followed by the period (.) or underscore (_) character:

    • Valid filename examples:
      c:\NewFolder\abc.log
      c:\NewFolder\0865842.log.txt
    • Invalid filename example:
      c:\NewFolder\logfile.txt

Auto-discovery rules

Each line where a supported time stamp is detected is a new log record. A line without a time stamp is considered as a continuation of an existing log record and appended to a line that contains a time stamp. Similarly, lines starting with whitespace characters, such as space or tab are treated as the log record continuation and appended. Time stamp, including date, time, timezone and offset, is searched in the first 64 characters of the log content, however this value is configurable. If offset or timezone are not found in the first 64 characters, the local host offset/timezone value is picked up. The supported time stamp formats include:

  • ISO 8601 format, '%Y-%m-%d %H:%M:%S`, for example 2022-04-17 11:25:12.345
  • RFC 3339 format, '%Y-%m-%dT%H:%M:%S', for example 2022-04-17T11:25:12.345
  • Unix Epoch format, providing the number of seconds that have elapsed since January 1, 1970 at midnight UTC time minus the leap seconds, for example: 1652088888
  • RFC 3164 format, '%b %d %H:%M:%S`, for example Apr 17 11:25:12
  • Db2 (IBM database 2) format, '%Y-%m-%d-%H.%M.%S', for example 2022-05-17-11.25.12.114000-300
  • IIS format, %m/%d/%Y, %H:%M:%S, for example: 04/17/2022, 11:25:12.345
  • W3C (World Wide Web Consortium) format, %Y-%m-%d %H:%M:%S, for example: "%Y-%m-%d %H:%M:%S" e.g.: 2022-04-17 11:25:12.345 UTC timezone
    • other common formats:

      • %d %b %Y %H:%M:%S, for example: 17 Apr 2022 11:25:12.345
      • %Y %b %d %H:%M:%S, for example: 2022 Apr 17 11:25:12.345
      • %d/%b/%Y:%H:%M:%S, for example.: 17/Apr/2022:11:25:12.345

JSON files are supported for Docker only. If the log tag is detected in a JSON file, then corresponding message is ingested and analyzed. Within the message, time stamp is searched. If unsupported time stamp is detected, the log file will not be analyzed or stored, but file status will be reported.

No auto-discovery

If you don't want Dynatrace to automatically discover new log files on a specific monitored host, set:

AppLogAutoDetection = false

On Linux:

  • Dynatrace OneAgent versions 103 and later:
    /var/lib/dynatrace/oneagent/agent/config/ruxitagentloganalytics.conf

  • Dynatrace OneAgent versions 102 and earlier:
    /var/lib/ruxit/agent/config/ruxitagentloganalytics.conf

On Windows:

  • Dynatrace OneAgent versions 103 and later:
    %PROGRAMDATA%\dynatrace\oneagent\agent\config\ruxitagentloganalytics.conf

  • Dynatrace OneAgent versions 102 and earlier:
    %PROGRAMDATA%\ruxit\agent\config\ruxitagentloganalytics.conf

OneAgent restart is not required.

What might prevent logs to appear on the server?

  • Detected over 200 rotated log file groups for a process.

    Dynatrace detects rotation scheme for log files and reports all log files under as a group under one name, which typically maps to many files on disk. Large number of rotated file groups typically means that Dynatrace did not recognize rotation pattern correctly and reports each physical file separately as a group. Once 200 reported rotated log file groups is reached, auto detection is turned off for this process. To resolve this issue, you can:

    • Define a custom file which properly captures variable parts of file name.
    • Push up the limit in agent configuration, FilesInGroup property (Log Monitoring configuration file).
  • Fast growing files are skipped.

    In a scenario where a log file grows very quickly (at a pace over 10 MB/s), it's content might be skipped. The agent will continue to send the log file as long as both, the network and the server, can handle such load. Note that 10 MB/s with typical compression is approximately 10 Mbps of upload traffic.

  • File name or path doesn't match typical log naming.

    The agent checks if logs match file name and path pattern that is typical for log files. If there is no match, the file will not be reported and sent to the server. This is needed to avoid false positives on detection of files as logs, and to prevent pulling non-log data from hosts. To remedy this you can set rules in the agent configuration, AutomaticFile property (Log Monitoring configuration file).

  • There are symbolic links in the file or the paths

    This limitation applies to custom files in which the user points to a path that contains symbolic links. The physical path of the file pointed by symbolic the link must meet criteria for a log. Otherwise, the symbolic links could be used to read non-log data from host.

  • File size is below 500 bytes.

Additional limitations for automatically detected files:

  • File cannot be deleted earlier than a minute after creation.
  • Files must be appended (old content is not uploaded).
  • Files must have text content.
  • Log files must be opened constantly (not just for short periods of adding log entry).
  • Log Files must be opened in write mode.