Create anomaly detection metric
In this use case, you need to automate anomaly detection. See how you can extract data from logs, create a processing rule, crate a metric and create an alert that will generate notification if an anomaly occurs.
Scenario
Your logs contain important business data in a raw format, which needs to be extracted and normalized before analysis. The log records that you ingest have the following structure:
"timestamp": "2023-08-10T14:30:23.383000000+02:00",
"content": "AddItemAsync called with userId=04e412f0-c030-4d4b-9c16-72250b071fbd, productId=2ZYFJ3GM2N, quantity=4",
"event.type": "LOG",
"host.name": "gke-myhost-cos-bd5b5ae9-sad319",
"k8s.container.name": "cartservice",
You plan to extract the product identifier and quantity already at ingest time with a log processing rule. Then turn it into a metric with the same dimensions. Then you can track this data via the metric. Finally, create an alert based on that metric values.
- Create a DQL query as an ad hoc analytics task to retrieve the product identifiers and quantities for each transaction.
- After you've verified that you have received the correct data, turn that log query into a metric. This will speed up your monitoring, and you can automate anomaly detection and keep your query costs under control.
- Create an alert based on the value of that metric.
Build DQL query
To build and run your query:
- In the Dynatrace menu, go to Logs.
- On the Logs and events page, turn on Advanced mode.
- Select
copy for the code sample below.
fetch logs | filter matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync") | parse content, "LD 'productId=' LD:product ', quantity=' INT:quantity" | fields timestamp, product, quantity | summarize sum(quantity), by:{product, bin(timestamp, 1h)} | fieldsRename field:`bin(timestamp, 1h)`, alias:`hour`
- Paste the query into the query edit box and select Run query.
This query performs the following actions:
- Retrieves logs for Kubernetes container named
cartservice
that contain a phraseAddItemAsync
in the log content. - Next, the result is parsed for product ID and quantity.
- These values are then presented in new fields:
product
andquantity
. - The
quantity
field is aggregated for eachproduct
per one-hour intervals. - The field containing the time for one-hour intervals is renamed to
hour
.
Create metric
To create a metric, you must first create a processing rule that will process the incoming log data and then create a metric based on log records matching the specific phrase. With each match, the processing rule will extract product ID and quantity values and place them in their respective fields. This will provide you with additional dimensions for this metric (productId
and quantity
).
Create log processing rule
- Go to Settings > Log Monitoring > Processing and select Add rule.
- In Rule name, give your processing rule a name:
Product and Quantity from logs
- Copy the filter command to the Matcher:
matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
- Copy the parse command to the Processor definition:
PARSE(content, "LD 'productId=' LD:product ', quantity=' INT:quantity")
- Select Save changes
This processing rule will process the incoming log data and extract the product id and quantity and place their values in their respective fields.
Add log metric
- Go to Settings > Log Monitoring > Metrics extraction and select Add log metric.
- In Key, append the metric name to the
log.
metric key:cartservice-product-quantity
- Add Matcher.
Use the DQL function for matching phrases, which is part of the Dynatrace Query Language (DQL):matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
- For the Metrc measurement option, select Attribute value.
- For Arttribute, enter
quantity
. - Select Add dimension and enter
product
. - Select Save changes to create the log metric.
Check the result in data explorer:
- Go to Data explorer.
- Query for metric
log.cartservice-product-quantity
- For the space aggregation function, select
sum
- For Split by, select
product
. - Select Run query to view the
quantity
value for eachproduct
graph. - Switch to the advanced mode and copy the metric selector query (
log.cartservice-product-quantity:splitBy(product):sum
).
Create alert
-
Go to Settings > Anomaly detection > Metric events and select Add metric event.
-
For Summary, enter
Abnormal product quantity
. -
In Query definition section:
- Select
Metric selector
option for Type. - Enter
log.cartservice-product-quantity:splitBy(product):sum
for Metric selector.
- Select
-
In Monitoring strategy section:
- Select Auto-adaptive threshold option for Model type.
- Switch on the Alert on missing data.
- Leave default value (
1
) for Number of signal fluctuations option. - Select Alert if metric is outside option for Alert condition.
-
In Event template section:
- Enter
Abnormal product quantity
for Title. - Modify the description to fit your needs or you can leave the default description format.
- Enter
-
Select Save changes to create the metric event.
To check the problem alert created by your anomaly detection metric:
- In the Dynatrace menu, go to Problems.
- Search for
Abnormal product quantity
problems.