Connecting log data to traces

Dynatrace is able to enrich your ingested log data with additional information that helps Dynatrace to recognize, correlate, and evaluate the data. Log enrichment results in a more refined analysis of your logs.

Log enrichment enables you to:

• Seamlessly switch context and analyze individual spans, transactions, or entire workloads
• Empower development teams by making it easier and faster for them to detect and pinpoint problems

Supported frameworks

Supported frameworks for trace/span log context enrichment:

• .NET (including Serilog framework and Microsoft Logging Extensions)
• Apache HTTP Server
• Go
• Java
• NGINX (automatic log enrichment is supported for error.logs, but manual log enrichment is required for access.logs)
• Node.js
• PHP

Supported frameworks for trace/span unstructured log context enrichment:

• .NET
• Java

There are two ways to enrich the log data that you send to Dynatrace:

• Automatic log enrichment
  This method is recommended for common technologies and applications generating structured log data.
• Manual log enrichment
  This method is recommended for custom technologies and applications generating unstructured log data.

Automatic log enrichment

You can enable log enrichment for a particular technology used to create log data and let Dynatrace automatically inject additional attributes into every log record received. This method is recommended for structured log data of known technologies.

To enable automatic log enrichment for a specific technology:

What does automatic log enrichment do?

Log enrichment modifies your ingested log data and adds the following information to each detected log record:

• dt.trace_id
• dt.span_id

Structured log data

For structured log data such as JSON, XML, and well-defined text log formats, Dynatrace adds an attribute field to the log record entry.

Example of enriched log data in JSON format

Log data in JSON format is enriched with additional <dt.trace_id> and <dt.span_id> properties.

jsoncopydownload
{
  "severity": "error",
  "time": 1638957438023,
  "pid": 1,
  "hostname": "paymentservice-788946fdcd-42lgq",
  "name": "paymentservice-charge",
  "dt.trace_id": "d04b42bc9f4b6ecdbf6bc9f4b6ecdbc",
  "dt.span_id": "9adc716eb808d428",
  "message": "Unsupported card type for cardNumber=************0454"
}

Example of enriched log data in XML format

Log data in XML format is enriched with additional <dt.trace_id> and <dt.span_id> nodes.

xmlcopydownload
<?xml version="1.0" encoding="windows-1252" standalone="no"?>
<record>
    <date>2021-08-24T14:41:36.565218700Z</date>
    <millis>1629816096565</millis>
    <nanos>218700</nanos>
    <sequence>0</sequence>
    <logger>com.apm.testapp.logging.jul.XMLLoggingSample</logger>
    <level>INFO</level>
    <class>com.apm.testapp.logging.jul.BaseLoggingSample</class>
    <method>info</method>
    <thread>1</thread>
    <message>Update completed successfully.</message>
    <dt.trace_id>513fcd4e9b08792fcd4e9b08792</dt.trace_id>
    <dt.span_id>125840e3125840e3</dt.span_id>
</record>

Unstructured log data

Unstructured log data is typically made of raw plain text that is sequentially ordered and is designed to be read by people. Dynatrace does not automatically enrich unstructured log data. Dynatrace is able to enrich unstructured log data, but appending additional information to log data may have an impact on third-party tools that consume that same log data.

Example of enriched log data in raw text format

Log data in raw text is enriched with additional [!dt dt.trace_id=$trace_id,dt.span_id=$span_id] string (attributes and their value).

copydownload
127.0.0.1 - [21/Oct/2021:10:33:28 +0200] GET /index.htm HTTP/1.1 404 597 [!dt dt.trace_id=aa764ee37ebaa764ee37eaa764ee37e,dt.span_id=b93ede8b93ede8]

Manual log enrichment

You can manually enrich your Dynatrace ingested log data by defining a log pattern to include the dt.span_id, dt.trace_id, and dt.trace_sampled fields.

Be sure to follow these rules for the format of the enriched fields in an unstructured log:

• Fields must be encapsulated in square brackets ([]) with a !dt prefix.
  For example, [!dt dt.trace_id=$trace_id,dt.span_id=$span_id]
• Fields must be formatted without double quotes.
• Any invalid characters for the field and field value must be escaped.
• Any control characters like \n must be excluded from the enrichment definition.

Example of manually enriching NGINX log data

Suppose you want to manually enrich your NGINX log data.

1. Define a log format in the NGINX configuration file:

copydownload
log_format custom '$remote_addr - [$time_local] $request $status $body_bytes_sent [!dt dt.trace_id=$trace_id,dt.span_id=$span_id]'; access_log logs/access.log custom

2. The result will be access.log file containing the enriched log records:

copydownload
127.0.0.1 - [21/Oct/2021:10:33:28 +0200] GET /index.htm HTTP/1.1 404 597 [!dt dt.trace_id=e1c0afeb0b8a91d7748139aa764ee37e,dt.span_id=e5e6748fab93ede8]
127.0.0.1 - [21/Oct/2021:10:33:31 +0200] GET /index.html HTTP/1.1 200 1056 [!dt dt.trace_id=81fe7816ba6c38f7aa09aef3684cd941,dt.span_id=3bdacc466ae073cd]

If you use a logging framework and log formatter that allows custom log patterns, you can adapt the pattern in the log formatter and directly access the Dynatrace enrichment attributes.

Example of manually enriching Log4j log data

In the Log4j PatternFormatter, you can specify a pattern like this to include Dynatrace enrichment information:

xmlcopydownload
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} dt.trace_id=%X{dt.trace_id} dt.span_id=%X{dt.span_id} - %msg%n"/>

Enriched log data analysis

• To check for a specific user of your application, use the log viewer and select the distributed trace link from a specific log record.
• You can view all logs for a particular user session to see how the user interacted with the application.
• With the Logs tab in distributed trace, you can navigate through the trace and, based on logs associated with that trace, quickly see what happened. For more about distributed trace analysis, see Log analysis with PurePath® technology

Disabling log enrichment

1. In the Dynatrace menu, go to Settings and select Server-side service monitoring > Deep monitoring.
2. Expand the New OneAgent features section.
3. On the Global tab, filter for enrichment.
4. Disable each log enrichment for each technology that you use to generate ingested log data.
5. Select Save changes to save your configuration.

Note: Disabling log enrichment in the technology you use results in empty/blank results for enriched functionalities, such as PurePath Log Analysis.