OpenTelemetry traces

A trace represents a single request as it is being handled by different services within an application. Distributed tracing is a method of observing requests as they propagate across distributed systems and micro-services, generating high-quality data about those requests and making them available for analysis. It shows the trace as it follows the flow of execution and can help teams understand how each involved service is performing.

Ingest trace data

You can ingest OpenTelemetry trace data (traces and spans) in two different ways:

Spans

Spans are created by the tracer and represent a single operation within the trace.

• A new span is created to represent operations on each individual service and the components involved.
• The span itself contains the context (metadata), which is a set of unique identifiers representing the request that each span is a part of.
• The span context is passed throughout the trace via context propagation.

A span usually contains:

• A span name
  • The span name should describe which operation is performed.
• Start and finish timestamps
• Attributes
  • Sets of key-value pairs.
• Events
  • A set of zero or more events, where each event is a tuple (timestamp, name, attributes)
• Parent span ID
  • The ID of the span above the current span.
  • If the current span is the root span, there is no parent span ID.
• Links to zero or more causally related spans.
• SpanContext
  • Context contains metadata and information required to reference a span.

Causal relationships between spans in a single trace

Tracer

The tracer is responsible solely for creating spans. It can be created and accessed through the TracerProvider class, which is also responsible for configuration.

Span context

The span context is a set of unique identifiers representing the request that each span is a part of.

• New spans are created to represent the work being done by the components and services involved in an application.
• The span context is passed throughout the trace via context propagation.

In contrast to the state of the span, the context represents the information that can identify a span within the trace and will be transferred to child spans across process boundaries. Thus, a context may contain:

• A trace ID, which identifies the trace itself. It is a unique 16-byte array with at least one non-zero byte. All spans within that trace carry the same Trace ID.
• A span ID, which identifies the span itself. It is a unique 8-byte array with at least one non-zero byte. When the context is passed to a child span, the span ID of the first span becomes the second span's parent span ID.
• Trace flags, which contain details about the trace. They are present in all traces.
• Trace state, which carries vendor-specific trace identification data in key-value pairs. Multiple tracing systems can participate in the same trace.

Attributes

Attributes can be assigned to a span or a resource:

• Span attributes are specific to each span (for example, span ID, port, thread ID).
• Resource attributes are carried across spans instead (for example, container ID, host name, deployment or pod name, process PID, and service name and version).
  • A resource is an immutable representation of any entity that generates telemetry data. All data sent to a backend must be associated with a resource to allow linking it with the appropriate entity.

You will see span attribute values and resource attributes values on your PurePath® distributed trace only if they have been configured in the allowlist.

Resource attributes can be set for each component producing telemetry data, and will, in turn, carry the same attributes. For example, if you have multiple calls to a currency conversion service through a shopping application, you might see the same resource attribute key service.name with a value of currencyservice present in each request made to that service. It will be the same each time the service is called within the trace, whereas span attributes on the same spans will differ each time.

Because spans can be created freely and their attributes can be customized, make sure to always follow the semantic conventions. This allows for telemetry collected from polyglot micro-service environments to be easily correlated and cross-analyzed.

Lifecycle

The basic lifecycle of a span is as follows.

1. A request is received by a service. The span context is extracted from the request headers if it exists.
2. A new span is created as a child of the extracted span context; if none exists, a new root span is created.
3. The service handles the request. Additional attributes and events are added to the span that are useful for understanding the context of the request, such as the hostname of the machine handling the request, or customer identifiers.
4. New spans may be created to represent work being done by subcomponents of the service.
5. When the service makes a remote call to another service, the current span context is serialized and forwarded to the next service by injecting the span context into the headers or message envelope.
6. The work being done by the service is completed successfully or not. The span status is appropriately set, and the span is marked as finished.