Generic log ingestion

Dynatrace version 1.213+

Generic log ingestion allows you to stream log records to the system. It is available only via Log Monitoring API v2 - POST ingest logs.

The generic log ingestion endpoint is located on ActiveGate.

  • The endpoint is enabled by default on all of your ActiveGates.
  • ActiveGate is responsible for serving the endpoint, collecting the data, and forwarding it to Dynatrace in batches.

ActiveGate will collect and attempt to automatically transform any log data containing the following elements:

  • Log content
  • Timestamp
  • Key-Values attributes

Log data transformation

Generic log ingestion automatically transforms status, severity, level, and syslog.severity severity keys to the loglevel attribute.

The input values for the status, severity, level, and syslog.severity severity keys are transformed (transformation is not case sensitive) into output values for the loglevel attribute based on the following mapping:

Input value Output value Example value
Begins with emerg or f EMERGENCY Emergency, fail, Failure
Begins with a ALERT alarm, Alert
Begins with c CRITICAL Critical, crucial
Begins with err ERROR Error, error
Begins with s SEVERE Severe, serious
Begins with w WARN warn, Warning
Begins with n NOTICE note, Notice
Begins with i INFO Info, information
Begins with d or trace or verbose DEBUG debug, TRACE, Verbose

Additionally, for each log event, a status attribute is created with a value that is a sum of loglevel values based on the following grouping:

Included loglevel values Combined status attribute value
SEVERE, ERROR, CRITICAL, ALERT, FATAL, EMERGENCY ERROR
WARN WARN
INFO, TRACE, DEBUG, NOTICE INFO
NONE NONE

For example: The level severity key in the generic log ingestion API request parameter contains the value serious.

  1. The level severity key is transformed into the loglevel attribute with the serious value mapped to SEVERE based on the above table.
  2. The loglevel attribute containing tje SEVERE value is grouped into status attribute. Based on the grouping table above, the status attribute will contain the ERROR value.
  3. For the log event details, the log viewer will report the following:
  • status - ERROR
  • loglevel - SEVERE

Log data queue

You can customize the log data queue properties by editing the custom.properties file (see Customize ActiveGate properties and parameters on your ActiveGate to set the following values:

[logdiskbuffer]
#disk_queue_path=<custom_path> # defaults to temp folder
#disk_queue_max_size_mb=<limit> # defaults to 300 MB

Limitations

Log Monitoring v2 has the following limits.

Log events per minute
Log data ingestion is limited to 10,000 log events per minute. If your log data stream carries more than 10,000 log events per minute, all log events above that number will be ignored and the displayed log data will contain the information that too many log events were ingested. This applies to all event sources (OneAgent and generic log ingestion).

Payload size
The maximum payload size of a single request is 1 MB.

Number of requests
The default maximum number of requests per minute is 10.

Log event minimum timestamp
The earliest timestamp for a log event is the current time minus 24 hours. This applies to all event sources (OneAgent and generic log ingestion). If the log event contains timestamp earlier than current time minus 24 hours, the event is dropped and the generic log ingestion API returns a 400 response code.

Log event maximum timestamp
The latest timestamp for a log event is the current time plus 12 hours. This applies to all event sources (OneAgent and generic log ingestion). If the log event contains timestamp later than current time plus 12 hours, the event is dropped and the generic log ingestion API returns a 400 response code.

Maximum attributes

A log event can have up to 50 attributes. Additional attributes (as they appear in the JSON stream) are ignored.

Example

In this example, the API request ingests log data that will create a log event with defined log attributes content, status, service.name, and service.namespace.

The API token is passed in the Authorization header.

The response contains response code 204.

Curl

curl -X POST \
   https://environment.activegate.domain.com/e/abc123a/api/v2/logs/ingest \
  -H 'Content-Type: application/json; charset=utf-8' \
  -H 'Authorization: Api-Token abcdefjhij1234567890' \
  -d '[
    {
    "content": "Exception: Custom error log sent via Generic Log Ingest",
    "status": "error",
    "service.name": "log-monitoring-tenant",
    "service.namespace": "dev-stage-cluster"
    }
]'

Request URL

https://environment.activegate.domain.com/e/abc123a/api/v2/logs/ingest

Response content

Success

Response code

204

FAQ

What is the limit of data that can be ingested?

Log data ingest is limited to 1,000 log events for trial accounts (automatically enabled) and 10,000 log events per minute for Dynatrace customers. This limit applies to log data provided by generic log ingestion and OneAgent. This can be increased upon requested.

Who can increase the ingestion limit per tenant?

Please contact a Dynatrace ONE product specialist by selecting the chat button in the upper-right corner of the Dynatrace menu bar.