powered by Grail
Dynatrace Log Management and Analytics can reshape incoming log data for better understanding, analysis, or further processing based on rules that you create.
Information can be logged in a very wide variety of formats depending on the application or process, operating system or other factors. Log Processing offers a central and flexible way of extracting value from those raw log lines.
Log processing occurs as log data arrives in the Dynatrace SaaS environment, and before it is written to disk (stored). You can process the log data once it reaches Dynatrace, by setting log processing rules. After the log data is processed, it is sent to storage and is available for further analysis. This method allows to process log data from all log ingest channels.
For example, you can extract numerical values from log lines with Log Processing, turn these into metrics on the Dynatrace Platform, and include them in dashboards and problem detection.
Log processing does not affect DDU consumption of log ingest.
Log processing is based on rules that contain a matcher and a processing rule definition.
- The matcher narrows down the available log data for executing this specific rule.
- The processing rule is a log processing instruction about how Dynatrace should transform or modify the log data from the matcher.
Log processing rules
In the Dynatrace menu, go to Settings > Log Monitoring > Log processing to view log processing rules that are in effect, reorder the existing rules, and create new rules. Rules are executed in the order in which they're listed, from top to bottom. This order is critical because a preceding rule may impact the log data that a subsequent rule uses in its definition.
Expand Details to examine a rule definition. A log processing rule consists of the following:
- Rule name
- Rule definition
You can turn any rule on or off in the Active column.
By default, log processing includes many enabled built-in rules responsible for cleaning up or normalizing log data. The name of every built-in rule starts with
You cannot modify these rules directly, but you have the ability to turn them off, copy their definitions and create new rules with your modifications.
To create a log processing rule
Select Add processing rule on the Log processing page.
Provide the name for the log processing rule.
Provide a log query in the Matcher section.
A log search query narrows down the available log data for executing this specific rule. Add a Matcher to your rule by pasting your matcher-specific DQL query.Matching based on previous rules is not supported
The Matcher operates on the initial data set before applying any processing rules. Matching records modified by preceding rules is not supported. For example, the modified field in rule 1 and used for matching in rule 2 will contain the original value for that field and will not use the modified field in rule 1.
Provide the processing rule definition.
The processing rule definition is a log processing instruction about how Dynatrace should transform or modify your log data.
The rule definition is created using log processing commands, functions and pattern matching (Dynatrace Pattern Language) that allows you to add, transform or remove incoming log records. This gives you total control over how your log data is presented to Dynatrace log monitoring.
Test the log processing rule.
1. Provide a log sample
You can test the rule definition by providing a fragment of the sample log manually in the Paste a log / JSON sample text box. Make sure it's in JSON format. Any textual log data should be inserted into the
contentfield of the JSON.
2. Run the test
Select Test the rule and view the result in the Test result text box.
Select Save changes.