• Home
  • How to use Dynatrace
  • Log Monitoring
  • Log data acquisition
  • Generic log ingestion

Generic log ingestion

Dynatrace version 1.213+

Generic log ingestion allows you to stream log records to the system. It is available only via Log Monitoring API - POST ingest logs.

The generic log ingestion endpoint is located on ActiveGate.

  • The endpoint is enabled by default on all of your ActiveGates.
  • ActiveGate is responsible for serving the endpoint, collecting the data, and forwarding it to Dynatrace in batches.
    • SaaS https://{your-environment-id}.live.dynatrace.com/api/v2/logs/ingest
    • Environment ActiveGate https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/logs/

ActiveGate will collect and attempt to automatically transform any log data containing the following elements:

  • Log content
  • Timestamp
  • Key-Values attributes

To list all predefined key-values attributes

  1. In the user menu, go to Environment API v2.
  2. Select Logs and expand the POST/logs/ingest endpoint.
  3. In the Parameters section, select the Schema tab and check the Supported semantic attribute keys section.

Log data transformation

Generic log ingestion automatically transforms status, severity, level, and syslog.severity severity keys to the loglevel attribute.

The input values for the status, severity, level, and syslog.severity severity keys are transformed (transformation is not case sensitive) into output values for the loglevel attribute based on the following mapping:

Input value Output value Example value
Begins with emerg or f EMERGENCY Emergency, fail, Failure
Begins with e excluding emerg ERROR Error, error
Begins with a ALERT alarm, Alert
Begins with c CRITICAL Critical, crucial
Begins with s SEVERE Severe, serious
Begins with w WARN warn, Warning
Begins with n NOTICE note, Notice
Begins with i INFO Info, information
Begins with d or trace or verbose DEBUG debug, TRACE, Verbose

Additionally, for each log event, a status attribute is created with a value that is a sum of loglevel values based on the following grouping:

Included loglevel values Combined status attribute value
SEVERE, ERROR, CRITICAL, ALERT, FATAL, EMERGENCY ERROR
WARN WARN
INFO, TRACE, DEBUG, NOTICE INFO
NONE NONE

For example: The level severity key in the generic log ingestion API request parameter contains the value serious.

  1. The level severity key is transformed into the loglevel attribute with the serious value mapped to SEVERE based on the above table.
  2. The loglevel attribute containing the SEVERE value is grouped into status attribute. Based on the grouping table above, the status attribute will contain the ERROR value.
  3. For the log event details, the log viewer will report the following:
  • status - ERROR
  • loglevel - SEVERE

Log data queue

You can customize the log data queue properties by editing the custom.properties file (see Configuration properties and parameters of ActiveGate on your ActiveGate to set the following values:

plaintext
[generic_ingest] #disk_queue_path=<custom_path> # defaults to temp folder #disk_queue_max_size_mb=<limit> # defaults to 300 MB
503 Usable space limit reached

The log data ingestion API returns a 503 Usable space limit reached error when the ingested log data exceeds the configured queue size. Typically, this is a temporary situation that occurs only during spikes. If this error persists, increase the value of disk_queue_max_size_mb in custom.properties to allow log ingestion spikes to be queued.

Example

In this example, the API request ingests log data that will create a log event with defined log attributes content, status, service.name, and service.namespace.

The API token is passed in the Authorization header.

The response contains response code 204.

Curl

shell
curl -X POST \ https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest \ -H 'Content-Type: application/json; charset=utf-8' \ -H 'Authorization: Api-Token dt0c01.abc123.abcdefjhij1234567890' \ -d '[ { "content": "Exception: Custom error log sent via Generic Log Ingest", "status": "error", "service.name": "log-monitoring-tenant", "service.namespace": "dev-stage-cluster" } ]'

Request URL

shell
https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest

Response content

plaintext
Success

Response code

204

Related topics
  • Log Monitoring API - POST ingest logs

    Push custom logs to Dynatrace via the Log Monitoring API v2.