Create anomaly detection metric

In this use case, you need to automate anomaly detection. See how you can extract data from logs, create a processing rule, crate a metric and create an alert that will generate notification if an anomaly occurs.

Scenario

Your logs contain important business data in a raw format, which needs to be extracted and normalized before analysis. The log records that you ingest have the following structure:

plaintextcopydownload
"timestamp": "2023-08-10T14:30:23.383000000+02:00",
"content": "AddItemAsync called with userId=04e412f0-c030-4d4b-9c16-72250b071fbd, productId=2ZYFJ3GM2N, quantity=4",
"event.type": "LOG",
"host.name": "gke-myhost-cos-bd5b5ae9-sad319",
"k8s.container.name": "cartservice",

You plan to extract the product identifier and quantity already at ingest time with a log processing rule. Then turn it into a metric with the same dimensions. Then you can track this data via the metric. Finally, create an alert based on that metric values.

1. Create a DQL query as an ad hoc analytics task to retrieve the product identifiers and quantities for each transaction.
2. After you've verified that you have received the correct data, turn that log query into a metric. This will speed up your monitoring, and you can automate anomaly detection and keep your query costs under control.
3. Create an alert based on the value of that metric.

Build DQL query

To build and run your query:

1. In the Dynatrace menu, go to Logs.
2. On the Logs and events page, turn on Advanced mode.
3. Select copy for the code sample below.

   dqlcopydownload
   fetch logs
   | filter matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")
   | parse content, "LD 'productId=' LD:product ', quantity=' INT:quantity"
   | fields timestamp, product, quantity
   | summarize sum(quantity), by:{product, bin(timestamp, 1h)}
   | fieldsRename field:`bin(timestamp, 1h)`, alias:`hour`

4. Paste the query into the query edit box and select Run query.

This query performs the following actions:

• Retrieves logs for Kubernetes container named cartservice that contain a phrase AddItemAsync in the log content.
• Next, the result is parsed for product ID and quantity.
• These values are then presented in new fields: product and quantity.
• The quantity field is aggregated for each product per one-hour intervals.
• The field containing the time for one-hour intervals is renamed to hour.

Create metric

To create a metric, you must first create a processing rule that will process the incoming log data and then create a metric based on log records matching the specific phrase. With each match, the processing rule will extract product ID and quantity values and place them in their respective fields. This will provide you with additional dimensions for this metric (productId and quantity).

Create log processing rule

1. Go to Settings > Log Monitoring > Processing and select Add rule.
2. In Rule name, give your processing rule a name:
   Product and Quantity from logs
3. Copy the filter command to the Matcher:

   dqlcopydownload
   matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")

4. Copy the parse command to the Processor definition:

   dqlcopydownload
   PARSE(content, "LD 'productId=' LD:product ', quantity=' INT:quantity")

5. Select Save changes

This processing rule will process the incoming log data and extract the product id and quantity and place their values in their respective fields.

Add log metric

1. Go to Settings > Log Monitoring > Metrics extraction and select Add log metric.
2. In Key, append the metric name to the log. metric key: cartservice-product-quantity
3. Add Matcher.
   Use the DQL function for matching phrases, which is part of the Dynatrace Query Language (DQL):

   dqlcopydownload
   matchesValue(k8s.container.name, "cartservice") and matchesPhrase(content, "AddItemAsync")

4. For the Metrc measurement option, select Attribute value.
5. For Arttribute, enter quantity.
6. Select Add dimension and enter product.
7. Select Save changes to create the log metric.

Check the result in data explorer:

1. Go to Data explorer.
2. Query for metric log.cartservice-product-quantity
3. For the space aggregation function, select sum
4. For Split by, select product.
5. Select Run query to view the quantity value for each product graph.
6. Switch to the advanced mode and copy the metric selector query (log.cartservice-product-quantity:splitBy(product):sum).

Create alert

1. Go to Settings > Anomaly detection > Metric events and select Add metric event.

2. For Summary, enter Abnormal product quantity.

3. In Query definition section:

   • Select Metric selector option for Type.
   • Enter log.cartservice-product-quantity:splitBy(product):sum for Metric selector.

   

4. In Monitoring strategy section:

   • Select Auto-adaptive threshold option for Model type.
   • Switch on the Alert on missing data.
   • Leave default value (1) for Number of signal fluctuations option.
   • Select Alert if metric is outside option for Alert condition.

   

5. In Event template section:

   • Enter Abnormal product quantity for Title.
   • Modify the description to fit your needs or you can leave the default description format.

   

6. Select Save changes to create the metric event.

To check the problem alert created by your anomaly detection metric:

1. In the Dynatrace menu, go to Problems.
2. Search for Abnormal product quantity problems.