Timestamp configuration

powered by Grail

Dynatrace 1.252+ OneAgent 1.247+

Dynatrace allows you to define rules that control log data timestamps.

Timestamp detection

By default, log monitoring automatically detects only the most common and unambiguous subset of date formats supported. For details, see Supported timestamp formats.

No timestamp detected

When Log Monitoring is unable to determine the time format, it treats each log line as a separate log entry with an automatically assigned timestamp (observation timestamp) using a one-minute time resolution, except for lines starting with whitespaces (space, tab), which are treated as a continuation of an entry.

Timestamp search limit

Regardless of format, the timestamp typically occurs within the first 64 characters of a log entry. However, the timestamp can occur elsewhere, in which case you can raise this limit on the OneAgent configuration page: Settings > Log Monitoring > OneAgent > Timestamp search limit.

Timestamp rules

Regardless of where it occurs in a log entry, a timestamp may be written in multiple formats. Dynatrace supports some timestamp formats by default, but sometimes multiple formats may fit the incoming log data and match the timestamp to an incorrect timestamp pattern.

Because of this, Log Monitoring also enables you to define a specific date format using timestamp rules that specify what should be considered a timestamp in a log record. These rules contain a timestamp pattern, time zone, and matchers.

• Pattern—Defines what should be considered a timestamp in your logline.
• Time zone—Defines the timestamp time zone. Optional if your timestamp pattern includes the timezone indicator (%z).
• Matcher—Narrows down the range for the rule and applies the timestamp pattern only to matched log entries. Because you can't use the log.content attribute in the timestamp pattern matchers, the highest granularity is a log source. Granularity is at this level because the timestamp pattern is used to split the contents of a log source into separate log records, so it is used before the log.content attribute's value (or any other attributes set on an individual log record's level) is determined.
  • If you create multiple rules matching the same log data, all defined time formats are searched for.
  • If you have at least one rule matching a given log, predefined formats are not applied to it.

Supported scopes

Three hierarchy scopes are supported: host, host group, and environment.

The hierarchy scopes are merged into one list in the following order:

1. Host rules
2. Host group rules
3. Environment rules

The OneAgent receives the merged list (merged lists from its respective hosts, host groups, and environments) with no indication of which scopes are defined.

Host scope

The host scope can be accessed through the Host settings for a specific host.

1. In the Dynatrace menu, go to Hosts and select your host.
2. Select More (…) > Settings to open the Host settings page.
3. On the Host settings page, select Log Monitoring > Timestamp configuration.
4. Configure data masking by adding rules with a set of matchers that specify what should be considered a timestamp in the log record.

Host group scope

The host group scope can be accessed via the Host page.

1. In the Dynatrace menu, go to Hosts and select your host.
2. In the Properties and tags section, select the Host group.
3. On the Settings page, select Log Monitoring > Timestamp configuration.
4. Configure data masking by adding rules with a set of matchers that specify what should be considered a timestamp in the log record.

Environment scope

The environment scope is available in the settings menu.

1. In the Dynatrace menu, go to Settings and select Log Monitoring > Timestamp configuration.
2. Configure data masking by adding rules with a set of matchers that identify your sensitive data.

Create rule

To add a rule (on the host, host group, or environment level) that interprets the incoming log data timestamps

1. Select Add rule to start configuring your rule.

2. Rule name
   The name to display for your configuration.

3. Pattern
   Enter the pattern to be read as a date from the logs. For details on timestamp formats, see Supported timestamp formats and the Date library.

   You need to specify at least the year, month, day, hours, minutes, and seconds, although you can use alternative formats for them. You can include the time zone indicator (%z) or specify the time zone separately in the rule definition.

4. Time zone
   Select the time zone to apply to this pattern.
   This setting is not enabled if you have already specified the timezone in the timestamp pattern (%z).

   You can select Local time zone to use the time zone of the host on which the OneAgent is running.

5. Select Add matcher to create a specific match for this rule and narrow down the scope for that rule.

   You can include multiple matchers in one rule. For example, the timestamp configuration rule can be applied to logs from a specific container, namespace, or log source. Multiple matchers with the same attribute use AND logic between matchers, while matchers with multiple values assigned to them use OR logic.

   Matcher attribute	Matching is based on
   Process group	The process group ID.
   Log source	A log path; wildcards are supported in form of an asterisk.
   K8s container name	The name of the Kubernetes container.
   K8s namespace name	The name of the Kubernetes namespace.
   K8s deployment name	The name of the Kubernetes deployment.
   Container name	The name of the container.
   DT entity container group ID	The DT entity container group ID.
   Process technology	The technology name.

6. Select the matching attribute.

7. Select Value and, from the Value list, select the detected log data items.

   You can add multiple values to the selected attribute. You can have one matcher that indicates the Log source and matches values /var/log/syslog and Windows Application Log. Use asterisks (*) as wildcards to get a partial match.

8. Select Save changes.

Rules are executed in the order in which they appear on the Timestamp configuration page.

Rules are executed in the order in which they appear on the Timestamp configuration page.

REST API

You can use the Settings API to manage your timestamp configuration:

• View schema
• List stored configuration objects
• View single configuration object
• Create new, edit, or remove existing configuration object

To check the current schema version for timestamp configuration, list all available schemas and look for the builtin:logmonitoring.timestamp-configuration schema identifier.

Timestamp configuration objects are available for configuration on the following scopes:

• environment—configuration object affects all hosts in a given environment.
• host_group—configuration object affects all hosts assigned to a given host group.
• host—configuration object affects only the given host.

To create a timestamp configuration using the API

1. Create an access token with the Write settings (settings.write) and Read settings (settings.read) permissions.

2. Use the GET a schema endpoint to learn the JSON format required to post your configuration. The timestamp configuration schema identifier (schemaId) is builtin:logmonitoring.timestamp-configuration. Here is an example JSON payload with the timestamp configuration:

   jsoncopydownload
   [
     {
       "insertAfter":"uAAZ0ZW5hbnQABnRlbmFudAAkMGUzYmY2ZmYtMDc2ZC0zNzFmLhXaq0",
       "schemaId": "builtin:logmonitoring.timestamp-configuration",
       "schemaVersion": "0.1.0",
       "scope": "tenant",
       "value": {
         "config-item-title": "Added from REST API",
         "date-time-pattern": "%Y-%m-%d %H:%M:%S",
         "timezone": "CET",
         "matchers": [
           {
             "attribute": "dt.entity.process_group",
             "operator": "MATCHES",
             "values": [
               "PROCESS_GROUP-05F00CBACF39EBD1"
             ]
           },
           {
             "attribute": "log.source",
             "operator": "MATCHES",
             "values": [
               "Windows System Log",
               "Windows Security Log"
             ]
           }
         ]
       }
     }
   ]

3. Use the POST an object endpoint to send your configuration.