8.2 Telemetry, Context Propagation, and the Observability Pipeline
An observability pipeline typically includes the application's API or SDK, auto-instrumentation, context propagation, data collection via an agent or gateway, processing (such as batching, filtering, sampling, or redaction), and one or more backend storage systems for metrics, logs, and traces.
Application SDK / auto-instrumentation
↓ OTLP
Agent/Gateway Collector
↓ batch/filter/sample/redact
Metrics · Logs · Trace backendsOpenTelemetry standardizes data generation, resource attributes, semantic conventions, and data transmission, but does not define a single, mandated storage backend.
Automatic and Manual Tracing in Coordination
Automatic tracing is well-suited for common boundaries such as HTTP requests, database operations, and message clients. Manual tracing, on the other hand, captures business steps, domain-level outcomes, and queue semantics.
Span names should have low cardinality (such as POST /orders/{id}/settlements) and should never directly include real URLs. Attributes should only record data that is necessary for diagnostics and within the scope of governance.
Context Must Propagate Across Synchronous and Asynchronous Boundaries
HTTP can use W3C traceparent/tracestate; messages should carry context in headers and extract it on the consumer side.
Do not blindly trust externally provided trace headers: validate their format, restrict baggage content, and decide at the trust boundary whether to retain or reconstruct the context. Baggage propagates downstream and should never contain keys or sensitive personal information.
Thread pools, futures, coroutines, and message callbacks may lose context, requiring framework integration or explicit attach/detach mechanisms. Leaking context to the next request poses the same risks.
Sampling Position and Visibility
- Head sampling determines visibility at the start of a trace, offering low cost and predictable performance, but may miss errors that occur later in the request;
- Tail sampling makes visibility decisions after collecting a complete or substantially complete trace, preserving error and slow-request data, but requires the Collector to buffer traces and consumes more resources;
- Probabilistic, rate limiting, and rule-based sampling can be combined for flexible control.
The sampling rate must be accounted for in metric interpretation. A trace being sampled should not skew the statistical completeness of SLO metrics.
Collector Has a Capacity Limit
Collector must include queueing, batching, retry mechanisms, memory limits, and persistent buffering strategies. When the backend is unavailable, it must not allow telemetry data to indefinitely consume disk space and memory on business nodes.
The Collector itself must be monitored: incoming and export rates, rejections, discards, queue length, oldest data age, retry behavior, and CPU and memory usage.
Telemetry Must Be Degradable
Telemetry collection or export failures must not block business requests. Synchronous logging to remote systems, forcing each request to flush trace data, or unbounded log buffering turn the observability system into a failure point.
Records that must be reliably preserved for audit purposes belong to audit business data and should be handled through a separate, reliable path. They must not be conflated with debug telemetry that operates on best-effort principles.
Version and Change Correlation
All signals must carry controlled resource attributes: service name, version, environment, zone/region, instance, and deployment identifier. Releases, configuration changes, and feature flag updates are recorded as events or annotations in the timeline.
This enables answering questions like, "Did the error rate spike only in version v42, zone-b, and among tenants that recently enabled the new algorithm?"
Chapter 12 will deepen coverage of budgeting, alerting, dashboards, RUM, and synthetic monitoring.
References
- OpenTelemetry, Specification Overview
- OpenTelemetry, Context Propagation
- OpenTelemetry Collector, Documentation