• Home
  • How to use Dynatrace
  • Log Monitoring
  • Log Monitoring configuration
  • Connecting log data to traces

Connecting log data to traces

Dynatrace version 1.232+

Dynatrace is able to enrich your ingested log data with additional information that helps Dynatrace to recognize, correlate, and evaluate the data. Log enrichment results in a more refined analysis of your logs.

Dynatrace version 1.232

Automatically connecting log data to traces works only for log data ingested via the API.

Dynatrace version 1.235

Automatically connecting log data to traces works for all log data, no matter how the log data was ingested by Dynatrace.

OneAgent version 1.239+

You can manually enrich logs for log data ingested by Dynatrace by defining a log pattern to include the dt.span_id, dt.trace_id, and dt.trace_sampled fields.

Log enrichment enables you to:

  • Seamlessly switch context and analyze individual spans, transactions, or entire workloads
  • Empower development teams by making it easier and faster for them to detect and pinpoint problems
I use Log Monitoring v1

To get the most out of log enrichment, you need to use the latest version of Dynatrace Log Monitoring.

Dynatrace deployments using Log Monitoring v1 can enrich your log data (log enrichment is performed by OneAgent), but Log Monitoring v1 doesn't recognize the additional information of your enriched log data, so you will not benefit from the enhanced analysis and reporting that is available with the latest version of Dynatrace Log Monitoring.

Supported frameworks

Supported frameworks for trace/span log context enrichment:

  • .NET (including Serilog framework and Microsoft Logging Extensions)
  • Apache HTTP Server
  • Go
  • Java
  • NGINX (automatic log enrichment is supported for error.logs, but manual log enrichment is required for access.logs)
  • Node.js
  • PHP

Supported frameworks for trace/span unstructured log context enrichment:

  • .NET
  • Java

There are two ways to enrich the log data that you send to Dynatrace:

  • Automatic log enrichment
    This method is recommended for common technologies and applications generating structured log data.
  • Manual log enrichment
    This method is recommended for custom technologies and applications generating unstructured log data.

Automatic log enrichment

You can enable log enrichment for a particular technology used to create log data and let Dynatrace automatically inject additional attributes into every log record received. This method is recommended for structured log data of known technologies.

Limiting log enrichment

Use Process group override to limit log enrichment to a specific process group or a process within a process group.

To enable automatic log enrichment for a specific technology:

  1. In the Dynatrace menu, go to Settings and select Server-side service monitoring > Deep monitoring.
  2. Scroll down to New OneAgent features and expand that section.
  3. On the Global tab, filter for enrichment.
  4. Enable each log enrichment for each technology that you use to generate ingested log data.
  5. Select Save changes to save your configuration.
  1. In the Dynatrace menu, go to Settings and select Server-side service monitoring > Deep monitoring.
  2. Scroll down to New OneAgent features and expand that section.
  3. On the Process group override tab, select Add process group override.
  4. Under Select the process group to be affected by settings, filter for the process group for which you want to enable OneAgent log enrichment.
  5. In the table of features for that process group, filter for enrichment and enable log enrichment for each technology that you use to generate ingested log data on the process group level.
  6. Select Add and then select Save changes.

What does automatic log enrichment do?

Log enrichment modifies your ingested log data and adds the following information to each detected log record:

  • dt.trace_id
  • dt.span_id

Structured log data

For structured log data such as JSON, XML, and well-defined text log formats, Dynatrace adds an attribute field to the log record entry.

Example of enriched log data in JSON format

Log data in JSON format is enriched with additional <dt.trace_id> and <dt.span_id> properties.

json
{ "severity": "error", "time": 1638957438023, "pid": 1, "hostname": "paymentservice-788946fdcd-42lgq", "name": "paymentservice-charge", "dt.trace_id": "d04b42bc9f4b6ecdbf6bc9f4b6ecdbc", "dt.span_id": "9adc716eb808d428", "message": "Unsupported card type for cardNumber=************0454" }

Example of enriched log data in XML format

Log data in XML format is enriched with additional <dt.trace_id> and <dt.span_id> nodes.

xml
<?xml version="1.0" encoding="windows-1252" standalone="no"?> <record> <date>2021-08-24T14:41:36.565218700Z</date> <millis>1629816096565</millis> <nanos>218700</nanos> <sequence>0</sequence> <logger>com.apm.testapp.logging.jul.XMLLoggingSample</logger> <level>INFO</level> <class>com.apm.testapp.logging.jul.BaseLoggingSample</class> <method>info</method> <thread>1</thread> <message>Update completed successfully.</message> <dt.trace_id>513fcd4e9b08792fcd4e9b08792</dt.trace_id> <dt.span_id>125840e3125840e3</dt.span_id> </record>

Unstructured log data

Important

Check if Dynatrace log enrichment has an impact on your existing log data pipeline before using automatic log enrichment on unstructured log data.

Unstructured log data is typically made of raw plain text that is sequentially ordered and is designed to be read by people. Dynatrace does not automatically enrich unstructured log data. Dynatrace is able to enrich unstructured log data, but appending additional information to log data may have an impact on third-party tools that consume that same log data.

Example of enriched log data in raw text format

Log data in raw text is enriched with additional [!dt dt.trace_id=$trace_id,dt.span_id=$span_id] string (attributes and their value).

127.0.0.1 - [21/Oct/2021:10:33:28 +0200] GET /index.htm HTTP/1.1 404 597 [!dt dt.trace_id=aa764ee37ebaa764ee37eaa764ee37e,dt.span_id=b93ede8b93ede8]

Manual log enrichment

OneAgent version 1.239+

You can manually enrich your Dynatrace ingested log data by defining a log pattern to include the dt.span_id, dt.trace_id, and dt.trace_sampled fields.

Be sure to follow these rules for the format of the enriched fields in an unstructured log:

  • Fields must be encapsulated in square brackets ([]) with a !dt prefix.
    For example, [!dt dt.trace_id=$trace_id,dt.span_id=$span_id]
  • Fields must be formatted without double quotes.
  • Any invalid characters for the field and field value must be escaped.
  • Any control characters like \n must be excluded from the enrichment definition.

Example of manually enriching NGINX log data

Suppose you want to manually enrich your NGINX log data.

  1. Define a log format in the NGINX configuration file:
log_format custom '$remote_addr - [$time_local] $request $status $body_bytes_sent [!dt dt.trace_id=$trace_id,dt.span_id=$span_id]'; access_log logs/access.log custom
  1. The result will be access.log file containing the enriched log records:
127.0.0.1 - [21/Oct/2021:10:33:28 +0200] GET /index.htm HTTP/1.1 404 597 [!dt dt.trace_id=e1c0afeb0b8a91d7748139aa764ee37e,dt.span_id=e5e6748fab93ede8] 127.0.0.1 - [21/Oct/2021:10:33:31 +0200] GET /index.html HTTP/1.1 200 1056 [!dt dt.trace_id=81fe7816ba6c38f7aa09aef3684cd941,dt.span_id=3bdacc466ae073cd]

If you use a logging framework and log formatter that allows custom log patterns, you can adapt the pattern in the log formatter and directly access the Dynatrace enrichment attributes.

Example of manually enriching Log4j log data

In the Log4j PatternFormatter, you can specify a pattern like this to include Dynatrace enrichment information:

xml
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} dt.trace_id=%X{dt.trace_id} dt.span_id=%X{dt.span_id} - %msg%n"/>

Enriched log data analysis

  • To check for a specific user of your application, use the log viewer and select the distributed trace link from a specific log record.
  • You can view all logs for a particular user session to see how the user interacted with the application.
  • With the Logs tab in distributed trace, you can navigate through the trace and, based on logs associated with that trace, quickly see what happened. For more about distributed trace analysis, see Log analysis with PurePath® technology

Disabling log enrichment

  1. In the Dynatrace menu, go to Settings and select Server-side service monitoring > Deep monitoring.
  2. Expand the New OneAgent features section.
  3. On the Global tab, filter for enrichment.
  4. Disable each log enrichment for each technology that you use to generate ingested log data.
  5. Select Save changes to save your configuration.

Note: Disabling log enrichment in the technology you use results in empty/blank results for enriched functionalities, such as PurePath Log Analysis.

Related topics
  • Log analysis with PurePath® technology

    Learn how to enhance your distributed traces analysis with logs.