8.1 Observability Signals: From User Symptoms to System State

Services are now reliably deployed on Kubernetes, yet users only report one thing: "Report generation is slow." Looking at just one machine's CPU usage or a single error log isn't enough to understand the full request path. When on-call engineers see a user complaint, how should they combine metrics, logs, and traces to narrow down the root cause?

Monitoring often checks known conditions; observability enables teams to interpret unknown internal states using system outputs. The goal isn't simply to "collect three types of data"; it's to shorten the path from user impact to actionable insight.

Start with User-Visible Outcomes

Prioritize using RED for service entry points:

• Rate: the rate of requests or tasks;
• Errors: the proportion of results that are invalid to users;
• Duration: the distribution of latency.

For underlying resources, use USE:

• Utilization: the proportion of resource utilization;
• Saturation: queue backlogs or over-subscription;
• Errors: resource-level errors.

These are not fixed dashboard templates. Queue consumers also need lag and the age of the oldest message, batch jobs require freshness and completion rates, and databases need transaction and replication signals.

Metrics Are Suitable for Aggregating Trends and Alerts

A Counter accumulates events monotonically within a run and can reset on events such as process restart; rate calculations must handle resets. A Gauge records a current measurement that can rise or fall, and a Histogram captures a distribution. Retain latency distributions to estimate percentiles and calculate SLO threshold ratios instead of relying only on averages.

For example, suppose a classic Histogram has a 300 ms bucket boundary and 8000 of 10,000 requests fall in that cumulative bucket. The proportion at or below 300 ms is 80%. Without that boundary, an estimate from adjacent buckets is not an exact threshold count; quantiles also depend on bucket precision. Do not average instance p99 values. Merge compatible distributions first, then estimate the quantile.

Labels create time-series dimensions. service, route, and status_code are typically under direct control; user_id, full URLs, and trace IDs introduce high cardinality and can overwhelm cost and query performance.

Avoid placing business IDs directly into metric labels. Use trace IDs or structured logging instead when you need request-level tracing.

Logs Record Discrete Events

Request-scoped structured logs should include timestamp, severity, service, version, environment, message, valid trace/span IDs, and a stable error code. Independent background events without a current span must not invent an ID. This is a custom application JSON log, requiring the collector to map resource attributes and context fields into the relevant log model. It is not an OTLP wire message.

{
  "timestamp": "2026-09-05T12:00:00Z",
  "severity": "ERROR",
  "service.name": "scoring",
  "service.version": "42.0.0",
  "deployment.environment.name": "production",
  "event": "settlement_failed",
  "message": "Settlement rejected because the score version changed",
  "matchId": "m-8472",
  "code": "VERSION_CONFLICT",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7"
}

Logs do not serialize entire objects or request bodies. Keys, tokens, personal information, and payment data must be automatically redacted by default, with access controls and retention policies enforced.

Traces Express a Path Across Boundaries of Work

A trace is composed of spans. A span records an operation, its parent-child or link relationships, timestamps, status, attributes, and events.

Asynchronous messages do not always maintain a strict parent-child relationship; instead, links can express that a consumption operation is triggered by multiple messages or message batches. Long-running business processes should not be forced into a single trace spanning days; instead, multiple traces can be connected via correlation IDs.

Three Signals Linked by Identity

SLO Alert: Error Rate Rising
  → Locate by service.version/zone
  → Use exemplar/trace to identify slow requests
  → Span traces point to database wait
  → Correlate with structured logs from the same instance

Consistent service.name, version, deployment environment, region, and trace context make those links possible. An exemplar attaches a trace reference to a particular metric observation; it does not create a trace-ID label. If the referenced trace was not retained or has expired, the link can still lead nowhere.

Observation Is Not Proof of Causation

A correlation between two metrics does not imply causation. Telemetry data can be sampled, lost, delayed, or incorrectly instrumented. Diagnosis requires combining change logs, experiments, business invariants, and underlying evidence.

The next lesson explains how to build a telemetry ingestion pipeline using OpenTelemetry and the Collector, without compromising business performance.

References

• Prometheus, Histograms and summaries

• Prometheus, Metric types

• OpenTelemetry, Deployment environment attributes

• OpenTelemetry, Concepts

• Google SRE Workbook, Monitoring