Custom log source

OneAgent version 1.251+ Dynatrace Cluster version 1.254+

In OneAgent version 1.249 and earlier, you need to add log files manually.

Custom log source configuration enables you to manually add log sources that have not been autodetected.

Autodiscovery might not identify a log source if a log file is not kept open for writing during a process.
It might also fail to find log sources that are not part of any processes or are part of short-lived processes.

In such cases, you can define, configure, and customize log sources to your needs.

Starting from OneAgent version 1.251, you can switch to the improved version of manual addition, which is the custom log source configuration. You can opt-in by selecting a banner on the Custom log source configuration page.

The entire process consists of two parts:

Source definition (custom log source configuration), which is described on this page.
Log acquisition (adding logs to storage), which is described on Log ingest rules.

If you need to store your custom logs, you need to complete both steps.

Advantages of custom log source configuration

Compared to the log addition in earlier versions of OneAgent versions, the process now has several improvements.

Custom log source configuration enables you to:

Define log sources without the process group context (adding a process group is optional). You can add up to three process groups to a log source.
Define rules within all three scopes (host, host group, environment).
Change security rules through files placed on the host.
Add a rule with multiple paths within one process group.
Use wildcards in directories.
Use a dedicated API.

Automatically migrate your legacy custom log source configuration: each of your existing rules is migrated to the environment scope with the corresponding process group context set accordingly. The names of migrated rules have the auto-migrated prefix.

Supported scopes

Three hierarchy scopes are supported: host, host group, and environment. The narrower a given scope, the higher its priority.

Log source rules configured for a host take precedence over log source rules configured for a host group.
Log source rules configured for a host group take precedence over log source rules configured for a Dynatrace environment.

Hosts

To configure custom log sources at the host level

Go to Hosts or Hosts Classic (latest Dynatrace) and select your host.
Select More (…) > Settings to open the Host settings page (available only on hosts assigned to a host group).
On the Host settings page, select Log Monitoring > Custom log sources.

Host groups

To configure custom log sources at the host group level

Go to Hosts or Hosts Classic (latest Dynatrace) and select your host.
Expand the Properties and tags section and select the Host group (available only on hosts assigned to a host group).
On the Host group settings page, select Log Monitoring > Custom log sources.

Environment

To configure custom log sources at the environment level

Go to Settings.
On the Settings page, select Log Monitoring > Custom log sources.

Configure log source

Go to the Custom log sources page at the host, host group, or environment level as described above.
Select Add custom log source and add Rule name.
optional Bind your rule to a process group by selecting the process group name from the dropdown menu.
In the Custom log source paths section, select Log source type. There are two source types available:
- Log path
- Windows Event Log Each rule is applied to one log source type. You can select either Log path or Windows Event Log.
To define a log path, select Add custom log source path, enter your path (for example, /var/lib/*.log or /var/log/sys.bin), and select Add Path. You can add up to 100 values per log source.

As in the following example, you can display the log name and path by right-clicking on the chosen event log and selecting Properties:
Select Save changes.
To activate your rule, turn on the Active toggle.

Set up permissions on Network File System (NFS)

When handling logs on NFS, access permissions are strictly enforced. This is true despite the increased capabilities of OneAgent, which allow it to access local files without having to read permissions for the dtuser user.

To allow OneAgent to process and ingest NFS-mounted resources:

Every non-local directory along the path needs to have at least the read and execute permissions set.
Every log file needs to have the read permission set.

Example log file to add:

/mnt/nfs/logs/app1/test.log
where /mnt/nfs is mounted to an external NFS resource.

In this scenario, both /mnt/nfs/logs and /mnt/nfs/logs/app1 need r-xpermissions for others, and every test.log.* file needs r-- permissions for others as shown below:

$ ls -l /mnt/nfs
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 logs

$ ls -l /mnt/nfs/log
drwxr-xr-x 3 1001 1002 4096 Sep 8 17:11 app1

$ ls -l /mnt/nfs/logs/app1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:22 test.log
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:23 test.log.1
-rw-rw-r-- 3 1001 1002 100 Jul 19 14:24 test.log.2

Log file matching

When configuring a custom log source, follow these rules:

Custom log paths must be absolute; relative paths are rejected. An absolute path has the following pattern:
- For Windows: any letter:\
- For Linux: Starts with /
A Windows Event Log path in Windows Event System must be a relative path.
Custom log sources can contain wildcards: # replaces a string of numbers, while * substitutes a string of any characters except for slash (/) or backslash (\). While * can be used both in file names and directories (starting from OneAgent version 1.251), # can be used only in file names.

Additionally, each custom log source path you add needs to be validated by OneAgent and abide by its security rules (file matching rules). The following security rules are applied on the OneAgent side:

Security rules

A log path is not in any of the following: /etc, /boot, /proc, /dev, /bin, /sbin, /usr, WindowsRoot:\windows, or WindowsRoot:\winnt. However, Windows|winnt\system32\winevt\Logs is accepted AND
A log path does not contain .ssh AND
A log path does not have the .pem extension AND
A log path is not located in a directory whose name starts with . (for example, /.hidden) AND
A log path must have the log extension separated by ., -, or _ (it can be followed by another extension with the same separator set) OR
- A log path must be located on the first or second level of the log or logs directory OR
- A log path must be located on any level of the /var/log directory OR
- A log path must have the file name catalina.out.

Files with paths that do not fulfill one or more criteria are not accepted. You can change the criteria by editing a configuration file on the host where OneAgent is installed.
Once the conditions above are met, log file matching takes place. Again, there are certain rules for log file matching.

Override security rules

You can add or override predefined security rules only in the configuration file on the host where OneAgent is installed.

Save your changes as a separate file placed in the OneAgent persistent configuration directory.
- /var/lib/dynatrace/oneagent/agent/config/logmodule on Linux and UNIX
- %PROGRAMDATA%\dynatrace\oneagent\agent\config\logmodule on Windows
  
  Any log file with the .json suffix is allowed in the above directories.
Do not edit the file that contains predefined rules:
- /opt/dynatrace/oneagent/agent/conf on Linux and UNIX
- %PROGRAMFILES%\dynatrace\oneagent\agent\conf on Windows
Rules defined by you under the custom configuration take precedence over the default rules. Additionally, the first matching rule determines whether a path passes the security test. The override configuration file (the one that you save in the persistent configuration directory) format needs to be the same as the format for a file with predefined rules.

Override configuration file

There is a predefined directory pattern that is executed from right to left. For example, /log/ will match /log/file and /var/log/file but not /log/dir/file
Only one directory is matched. For example, /log/*/ will match /log/dir/file but not /log/dir/dir2/file
The [-.\\_] expression in a pattern means that one of the characters provided in the square brackets must be present for a match to occur.

Example override configuration file

{
    "@version":"1.0.0",
    "allowed-log-paths-configuration":[
       {
          "directory-pattern":"/",
          "file-pattern":"*.pem",
          "action":"EXCLUDE" // or INCLUDE
       },
       ... your rules ...
    ]
 }

Examples of OneAgent security rules

Each custom log source path you add needs to be validated by OneAgent and abide by its security rules (file matching rules). Some predefined security rules are applied on the OneAgent side. Examples of exclude and include rules for UNIX, Linux, and Windows are listed in the table below.

Operating system	Directory pattern	File pattern	Action
UNIX	`/`	`*.pem`	EXCLUDE
UNIX	`/`	`[-.\\_]log[-.\\_]`	INCLUDE
Linux	`/.ssh/`	*	EXCLUDE
Linux	`/`	`*[-.\\_]log`	INCLUDE
Windows	`/.*/`	*	EXCLUDE
Windows	`/windows/system32/winevt/Logs/`	*	INCLUDE

Security rule lists for UNIX, Linux, and Windows

The full list of security rules for UNIX:

{
   "@version":"1.0.0",
   "allowed-log-paths-configuration":[
      {
         "directory-pattern":"/",
         "file-pattern":"*.pem",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.ssh/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":".*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/etc/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/boot/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/proc/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/dev/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/bin/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/sbin/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/usr/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log[-.\\_]*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"catalina.out*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/lib/docker/containers/*/",
         "file-pattern":"*.log",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      }
   ]
}

The full list of security rules for Linux:

{
   "@version":"1.0.0",
   "allowed-log-paths-configuration":[
      {
         "directory-pattern":"/",
         "file-pattern":"*.pem",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.ssh/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":".*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/etc/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/boot/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/proc/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/dev/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/bin/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/sbin/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/usr/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log[-.\\_]*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"catalina.out*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/lib/docker/containers/*/",
         "file-pattern":"*.log",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/var/log/*/*/*/*/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      }
   ]
}

The full list of security rules for Windows:

{
   "@version":"1.0.0",
   "allowed-log-paths-configuration":[
      {
         "directory-pattern":"/",
         "file-pattern":"*.pem",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.ssh/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/.*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":".*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/windows/system32/winevt/Logs/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/winnt/system32/winevt/Logs/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/windows/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/winnt/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/windows/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/winnt/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/windows/*/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/winnt/*/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/windows/*/*/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/winnt/*/*/*/",
         "file-pattern":"*",
         "action":"EXCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log[-.\\_]*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"*[-.\\_]log",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/",
         "file-pattern":"catalina.out*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/log/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/",
         "file-pattern":"*",
         "action":"INCLUDE"
      },
      {
         "directory-pattern":"/logs/*/",
         "file-pattern":"*",
         "action":"INCLUDE"
      }
   ]
}