Generic log ingestion
Generic log ingestion allows you to stream log records to the system. It is available only via Log Monitoring API - POST ingest logs.
The generic log ingestion endpoint is located on ActiveGate.
- The endpoint is enabled by default on all of your ActiveGates.
- ActiveGate is responsible for serving the endpoint, collecting the data, and forwarding it to Dynatrace in batches.
- SaaS https://{your-environment-id}.live.dynatrace.com/api/v2/logs/ingest
- Environment ActiveGate https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/logs/
ActiveGate will collect and attempt to automatically transform any log data containing the following elements:
- Log content
- Timestamp
- Key-Values attributes
To list all predefined key-values attributes
- In the user menu, go to Environment API v2.
- Select Logs and expand the POST/logs/ingest endpoint.
- In the Parameters section, select the Schema tab and check the Supported semantic attribute keys section.
Log data transformation
Generic log ingestion automatically transforms status
, severity
, level
, and syslog.severity
severity keys to the loglevel
attribute.
The input values for the status
, severity
, level
, and syslog.severity
severity keys are transformed (transformation is not case sensitive) into output values for the loglevel
attribute based on the following mapping:
Input value | Output value | Example value |
---|---|---|
Begins with emerg or f | EMERGENCY | Emergency , fail , Failure |
Begins with e excluding emerg | ERROR | Error , error |
Begins with a | ALERT | alarm , Alert |
Begins with c | CRITICAL | Critical , crucial |
Begins with s | SEVERE | Severe , serious |
Begins with w | WARN | warn , Warning |
Begins with n | NOTICE | note , Notice |
Begins with i | INFO | Info , information |
Begins with d or trace or verbose | DEBUG | debug , TRACE , Verbose |
Additionally, for each log event, a status
attribute is created with a value that is a sum of loglevel
values based on the following grouping:
Included loglevel values | Combined status attribute value |
---|---|
SEVERE , ERROR , CRITICAL , ALERT , FATAL , EMERGENCY | ERROR |
WARN | WARN |
INFO , TRACE , DEBUG , NOTICE | INFO |
NONE | NONE |
For example:
The level
severity key in the generic log ingestion API request parameter contains the value serious
.
- The
level
severity key is transformed into theloglevel
attribute with theserious
value mapped toSEVERE
based on the above table. - The
loglevel
attribute containing theSEVERE
value is grouped intostatus
attribute. Based on the grouping table above, thestatus
attribute will contain theERROR
value. - For the log event details, the log viewer will report the following:
- status -
ERROR
- loglevel -
SEVERE
Log data queue
You can customize the log data queue properties by editing the custom.properties
file (see Configuration properties and parameters of ActiveGate on your ActiveGate to set the following values:
[generic_ingest]
#disk_queue_path=<custom_path> # defaults to temp folder
#disk_queue_max_size_mb=<limit> # defaults to 300 MB
The log data ingestion API returns a 503 Usable space limit reached
error when the ingested log data exceeds the configured queue size. Typically, this is a temporary situation that occurs only during spikes. If this error persists, increase the value of disk_queue_max_size_mb
in custom.properties
to allow log ingestion spikes to be queued.
Example
In this example, the API request ingests log data that will create a log event with defined log attributes content
, status
, service.name
, and service.namespace
.
The API token is passed in the Authorization header.
The response contains response code 204
.
Curl
curl -X POST \
https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest \
-H 'Content-Type: application/json; charset=utf-8' \
-H 'Authorization: Api-Token dt0c01.abc123.abcdefjhij1234567890' \
-d '[
{
"content": "Exception: Custom error log sent via Generic Log Ingest",
"status": "error",
"service.name": "log-monitoring-tenant",
"service.namespace": "dev-stage-cluster"
}
]'
Request URL
https://environment.activegate.domain.com:9999/e/abc123a/api/v2/logs/ingest
Response content
Success
Response code
204