8.2 Instrumentation, Context Propagation, and Telemetry Pipelines

An observability pipeline typically includes the application's API or SDK, auto-instrumentation, context propagation, data collection via an agent or gateway, processing (such as batching, filtering, sampling, or redaction), and one or more backend storage systems for metrics, logs, and traces.

Application SDK / auto-instrumentation
              ↓ OTLP
Agent/Gateway Collector
   ↓ batch/filter/sample/redact
Metrics · Logs · Trace backends

OpenTelemetry standardizes data generation, resource attributes, semantic conventions, and data transmission, but does not define a single, mandated storage backend.

Combine Automatic and Manual Instrumentation

Automatic instrumentation suits for common boundaries such as HTTP requests, database operations, and message clients. Manual instrumentation captures business steps, domain-level outcomes, and queue semantics.

Span names should have low cardinality (such as POST /orders/{id}/settlements) and should never directly include real URLs. Attributes should only record data that is necessary for diagnostics and within the scope of governance.

Context Must Propagate Across Synchronous and Asynchronous Boundaries

HTTP can use W3C traceparent/tracestate; messages carry context in headers for extraction by consumers. A version-00 traceparent can be 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Its trace ID has 32 hexadecimal characters and parent ID has 16; neither may be all zero. The parent ID identifies the sending span in propagated context, while the receiver creates its own new span. The 01 sampled flag also does not guarantee that the backend retains a complete trace.

Do not blindly trust externally provided trace headers: validate their format, restrict baggage content, and decide at the trust boundary whether to retain or reconstruct the context. Baggage propagates downstream and should never contain keys or sensitive personal information.

Thread pools, futures, coroutines, and message callbacks may lose context, requiring framework integration or explicit attach/detach mechanisms. Leaking context to the next request poses the same risks.

Sampling Position and Visibility

• Head sampling determines visibility at the start of a trace, offering low cost and predictable performance, but may miss errors that occur later in the request;
• Tail sampling decides from spans received within a waiting window, allowing priority for errors and slow requests. Long traces or late spans can exceed that window, and buffering requires additional Collector resources;
• Probabilistic, rate limiting, and rule-based sampling can be combined for flexible control.

Spans belonging to one trace usually need routing by trace ID to the same tail-sampling instance. Otherwise each instance may see only part of the trace. Tail sampling cannot recover spans already discarded by head sampling, so the two strategies cannot be combined merely because each advertises error retention.

SLO counts should come from a metric path covering all relevant requests. Tail-sampling by error or latency before deriving an error rate from remaining spans biases the result toward failed and slow requests. Dividing by one sampling rate cannot correct that selection bias. The independent metric path still needs monitoring for data loss and export failure.

Collector Has a Capacity Limit

Collector must include queueing, batching, retry mechanisms, memory limits, and persistent buffering strategies. When the backend is unavailable, it must not allow telemetry data to indefinitely consume disk space and memory on business nodes.

A persistent queue retains only data successfully enqueued, and still has disk-capacity and retry-duration limits. It cannot recover spans never exported by the SDK or automatically provide end-to-end exactly-once delivery. The ordering of memory limiting, batching, sampling, and redaction also affects peak resource use and exposure of sensitive data; validate it with the configuration.

The Collector itself must be monitored: incoming and export rates, rejections, discards, queue length, oldest data age, retry behavior, and CPU and memory usage.

Telemetry Must Be Degradable

Telemetry collection or export failures must not block business requests. Synchronous logging to remote systems, forcing each request to flush trace data, or unbounded log buffering turn the observability system into a failure point.

Records that must be reliably preserved for audit purposes belong to audit business data and should be handled through a separate, reliable path. They must not be conflated with debug telemetry that operates on best-effort principles.

Version and Change Correlation

All signals must carry controlled resource attributes: service name, version, environment, zone/region, instance, and deployment identifier. Releases, configuration changes, and feature flag updates are recorded as events or annotations in the timeline.

This supports questions such as "Did errors rise only in v42, zone-b, and tenants with the new algorithm enabled?" Versions, zones, and a bounded set of feature-flag states can support aggregation. Investigate individual tenants through controlled logs or traces instead of adding every tenant ID to long-lived metric labels.

Chapter 12 covers cardinality budgets, alerting, dashboards, RUM, and synthetic monitoring.

References

• W3C, Trace Context

• OpenTelemetry, Sampling

• OpenTelemetry, Scaling the Collector

• OpenTelemetry, Collector resiliency

• OpenTelemetry, Specification Overview

• OpenTelemetry, Context Propagation

• OpenTelemetry Collector, Documentation