Log storage configuration

powered by Grail

Dynatrace version 1.252+ OneAgent version 1.243+

Dynatrace allows you to include and exclude specific log sources for your analysis. Using Dynatrace identity and access management (IAM) framework, you can control which user can change configurations on which scope.

The configuration is based on rules that use matchers for hierarchy, log path, and process groups. These rules determine which log files among those detected by OneAgent, either automatically or defined as custom log sources, are ingested.

Log ingest rule

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log storage configuration.

2. Select Add rule and provide the name for your configuration.
   By default, the Include in storage button is turned on, indicating that items configured by this rule will be stored in Dynatrace. Alternatively, you can select the Exclude from storage rule type.

3. Expand Details of your new rule and select Add matcher to create a specific match for this rule.
   Multiple matchers can be included in one rule.

   Other than the Log source attribute in Windows (due to file paths being case insensitive), matchers are case-sensitive.

4. Select the matching attribute:

If no wildcard is used in the value, then the matcher looks for an exact fit to the value. If a wildcard is used, the matcher looks for the exact match. For example, the value INFO results in sending only the log data having the exact INFO string, but the value *INFO* (using the wildcards) matches log data that contain the INFO string in its content.

5. Select Add value and, from the Values, select the detected log data items (log files or process groups that contain log data). Multiple values can be added to the selected attribute. You can have one matcher that indicates log source and matches values /var/log/syslog and Windows Application Log.

6. Save changes.

Defined rules can be reordered and are executed in the order in which they appear on the Log storage page.

7. To activate your rule, turn on the Active toggle.

Matching a list of rules to log data

Matching occurs in a predefined hierarchy and rules are executed from top to bottom. This means that if a rule above on the list matches certain log data, then the lower ones will be omitted. Items matched in the higher-level configurations are overwritten in the lower-level configurations if they match the same log data. If no rule is matched, the file is not sent. The matching hierarchy is as follows:

1. Host configuration rules
2. Host group configuration rules
3. Tenant configuration rules

Configuration scopes

Three hierarchy scopes are supported: host, host group, and tenant. The scope with the least possible set of rules has priority over larger sets.

1. Log storage rules configured for a host take precedence over log storage rules configured for a host group.
2. Log storage rules configured for a host group take precedence over log storage rules configured for a tenant.

Host scope

The host scope can be accessed through the Host settings for a specific host.

1. In the Dynatrace menu, go to Hosts.
2. Find and select your host to display the host overview page.
3. In the upper-right corner of the host overview page, select More (…) > Settings.

4. From the host settings, go to Log Monitoring > Log storage configuration.
5. Configure storage upload by adding rules with a set of attributes that matches the log data to be stored by Dynatrace.

Host group scope

The host group scope can be accessed via the Host page.

1. In the Dynatrace menu, go to Hosts and select the host that interests you.
2. On the host overview page, select Properties and tags.
3. On the Properties and tags panel, find the Host group property to see the name of the host group to which the selected host belongs.

   The Host group property is not displayed when the selected host doesn't belong to any host group.

4. Select the host group name to list all hosts in that host group. This displays the OneAgent deployment page filtered by the selected host group. Each listed host has a Host group: <group name> link, where <group name> is the name of the host group that you want to configure.
5. Select the host group name in any row.

6. In the host group settings, select Log Monitoring > Log storage configuration.
7. Configure storage upload by adding rules with a set of attributes that matches the log data to be stored by Dynatrace.

Tenant scope

The tenant scope is available in the settings menu.

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log storage configuration.
2. Configure storage upload by adding rules with a set of attributes that matches the log data to be stored by Dynatrace.

List hosts and host groups with overriding rules

The table on Settings > Log Monitoring > Log storage configuration lists all log storage rules that you have set at the tenant level. However, you may want to see where you have set log storage rules for hosts and host groups that override the tenant-level rules.

To list all entities (hosts and host groups) to which more specific log storage rules are applied

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log storage configuration.

2. In the upper-right corner of the Log storage configuration page, select More (…) > Hierarchy and overrides. A searchable Hierarchy and overrides panel lists all entities (hosts and host groups) on which you have set log storage rules that override the tenant-level rules listed on Settings > Log Monitoring > Log storage configuration.

3. Select an entity name to go to that entity's Log storage configuration page.

Example upload

In this example, we configure the tenant storage upload for c:\inetpub\logs\LogFiles\ex_*.log files in two process groups: IIS (PROCESS_GROUP-3D9D854163F8F07A) and IIS (PROCESS_GROUP-4A7B47FDB53137AE). The log storage rule consists of two matchers: the first matcher finds the process groups and the second matcher matches only for the defined log source.

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log storage configuration.
2. Select Add rule and provide the title for your configuration.
3. Select Add matcher. This is the first matcher to match two specified process groups.
4. From the Attribute list, select Process group.
5. Select Add value and type IIS, and then, from the suggestion list, select IIS (PROCESS_GROUP-3D9D854163F8F07A).
6. Select Add value again, type IIS and select the second process group from the suggestion list: IIS (PROCESS_GROUP-4A7B47FDB53137AE).
7. Select Add matcher again. This is the second matcher to match the specified log data source.
8. From the Attribute list, select Log source.
9. Select Add value and enter c:\inetpub\logs\LogFiles\ex_*.log as the value.
10. Save changes.

Example exclude

In this example, we configure the tenant storage upload for all log sources except c:\inetpub\logs\LogFiles\ex_*.log files in a process group IIS (PROCESS_GROUP-4A7B47FDB53137AE).

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Log storage configuration.
2. Select Add rule and provide the title for your configuration.
3. Turn off Send to storage.
4. Select Add matcher. This is the first matcher to match the specified process group.
5. From the Attribute list, select Process group.
6. Select Add value and type IIS, and then, from the suggestion list, select IIS (PROCESS_GROUP-3D9D854163F8F07A).
7. Select Add matcher again. This is the second matcher to exclude the specified log data source.
8. From the Attribute list select Log source.
9. Select Add value and enter c:\inetpub\logs\LogFiles\ex_*.log as a value.
10. Save changes.

REST API

You can use the Settings API to manage your log storage configuration:

• View schema
• List stored configuration objects
• View single configuration object
• Create new, edit, or remove existing configuration object

To check the current schema version for log storage configuration, list all available schemas and look for the builtin:logmonitoring.log-storage-settings schema identifier.

Log storage configuration objects are available for configuration on the following scopes:

• tenant – configuration object affects all hosts on a given tenant.
• host_group – configuration object affects all hosts assigned to a given host group.
• host – configuration object affects only the given host.

To create a log storage configuration using the API:

1. Create an access token with the Write settings (settings.write) and Read settings (settings.read) permissions.

2. Use the GET a schema endpoint to learn the JSON format required to post your configuration. The log storage configuration schema identifier (schemaId) is builtin:logmonitoring.log-storage-settings. Here is an example JSON payload with the log storage configuration:

   jsoncopydownload
   [
     {
       "insertAfter":"uAAZ0ZW5hbnQABnRlbmFudAAkMGUzYmY2ZmYtMDc2ZC0zNzFmLhXaq0",
       "schemaId": "builtin:logmonitoring.log-storage-settings",
       "schemaVersion": "0.1.0",
       "scope": "tenant",
       "value": {
         "config-item-title": "Added from REST API",
         "send-to-storage": true,
         "matchers": [
           {
             "attribute": "dt.entity.process_group",
             "operator": "MATCHES",
             "values": [
               "PROCESS_GROUP-05F00CBACF39EBD1"
             ]
           },
           {
             "attribute": "log.source",
             "operator": "MATCHES",
             "values": [
               "Windows System Log",
               "Windows Security Log"
             ]
           }
         ]
       }
     }
   ]

3. Use the POST an object endpoint to send your configuration.

Examples

The examples that follow show the results of various combinations of rules and matchers.

Example 1: Multiple rules

In this example, there are two rules:

• Rule 1 is an Exclude rule and has two matchers: the process group attribute is Apache, and the Log source attribute is access.log).
• Rule 2 is an Include rule and has one matcher: the process group attribute is Apache.

Results: access.log is not sent, error.log (of Apache) is sent, and error.log (of other PG) is not sent.

• access.log written by Apache matches the first rule, which has send-to-storage: false, so it is not sent.
• access.log not written by Apache doesn't match the first rule (due to incorrect process group), and doesn't match the second rule, so it is not sent.
• error.log written by Apache does not match the first rule (due to incorrect source), but it matches the second rule, which has send-to-storage: true, so it is sent.
• error.log not written by Apache doesn't match the first rule (due to both incorrect process group and log source), and doesn't match the second rule, so it is not sent.

jsoncopydownload
{
  "send-to-storage": false,
  "matchers": [
    {
      "attribute": "log.source",
      "values": [
        "/path/to/access.log"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 2: Send logs written by Apache and containing 'ERROR'

This task requires setting one rule with two matchers.

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 3: Send logs written by Apache or containing 'ERROR'

This task requires setting two rules with one matcher each.

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 4: Send logs written by Apache, and containing 'ERROR' and 'Customer'

This task requires setting one rule with three matchers, with one value each.

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*"
      ]
    },
    {
      "attribute": "log.content",
      "values": [
        "*Customer*"
      ]
    },
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 5: Send logs written by Apache, and containing 'ERROR' or 'Customer'

This task requires setting one rule with two matchers: a matcher with the process group value, and a matcher with two content values.

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "log.content",
      "values": [
        "*ERROR*", "*Customer*"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Example 6: Send logs written by Apache or MySQL

This task requires setting two rules, or one rule with one matcher having two values.
Rules with two matchers will not work here.

Setting two rules:

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID"
      ]
    }
  ],
  "enabled": true
}

Setting one rule with one matcher having two values:

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
}

Example 7: Send all logs

This task requires setting a rule without any matchers.

jsoncopydownload
{
  "send-to-storage": true,
  "matchers": [
  ],
  "enabled": true
}

Example 8: Send all logs except Apache and MySQL logs

This task requires setting two rules.

• The first rule is an Exclude rule with one matcher having two values.
• The second rule does not contain any matchers.

The rules have to be executed in the order indicated below.

jsoncopydownload
{
  "send-to-storage": false,
  "matchers": [
    {
      "attribute": "dt.entity.process_group",
      "values": [
        "PROCESS_GROUP-APACHEID", "PROCESS_GROUP-MYSQL"
      ]
    }
  ],
  "enabled": true
},
{
  "send-to-storage": true,
  "matchers": [
  ],
  "enabled": true
}

Frequently asked questions