12.3 Active Observability and Observability as Code

Server-side metrics tell you what’s happening inside your system, but they don’t necessarily reflect what users actually experience. Browser rendering, DNS resolution, CDN behavior, third-party scripts, carrier network conditions, and regional routing issues can all cause users to fail tasks even when server logs show perfect health.

RUM Observes Real User Distribution

Real User Monitoring (RUM) collects experience signals from actual browser or client sessions, such as navigation times, key interactions, resource loading, frontend errors, and geographic or device dimensions.

The advantages of RUM are its coverage of real-world environments and long-tail scenarios, but it comes with clear limitations:

• When traffic is low or users are absent in certain regions, there aren't enough samples to draw meaningful insights;
• Ad blockers, privacy settings, and network failures can prevent telemetry from reaching the system;
• Dimensions like users, URLs, devices, and sessions can lead to high cardinality and privacy risks;
• Averages can mask issues such as poor network conditions, low-end devices, or underrepresented regions.

When designing RUM systems, use aggregated routes, coarse-grained geographic segmentation, and controlled device categorization. Minimize, sample, limit retention periods, and enforce access controls on any data that could identify individuals. Never treat query strings, email addresses, or full user IDs as convenient diagnostic tags.

Synthetic Active Validation Key Journey

Synthetic monitoring is executed by controlled probes that perform fixed operations at regular intervals:

DNS resolution → TLS handshake → homepage load → login → search → place order → validate result

It can continuously validate without relying on real user traffic and is well-suited for establishing external perspectives from multiple geographic regions. Probes should be layered: high-frequency, low-cost HTTP/TCP checks cover entry points; lower-frequency browser-based journeys validate critical business workflows.

Synthetic accounts and data must be isolated, with operations designed to be idempotent or easily cleanable, to prevent contamination of production orders and financial reports. Probes rely on their own runner, DNS resolution, and credentials, so it's essential to distinguish between "product failure" and "probe infrastructure failure".

RUM, Synthetic, and Server-Side Telemetry Complement Each Other

Signal	Best at Answering	Key Limitations
Server-side telemetry	Why services are slow or failing internally	Cannot capture browser, client, or end-user experience
RUM	Which user journeys are actually impacted in real-world scenarios	No data during low-traffic periods; subject to sampling bias
Synthetic	Whether predefined user journeys are available from specific starting points	Only covers pre-defined paths and workflows

A regional outage can first be detected by synthetic monitoring, then validated by RUM to determine the actual scope of impact, and finally pinpointed using traces, logs, and metrics to identify specific dependencies. These three telemetry sources should share stable semantic contexts (such as service, environment, and release) and be linked via trace IDs or exemplar data, rather than forcing all events into a single monolithic storage model.

Observability as Code: Managing Executable Knowledge

What's versioned shouldn't be limited to dashboard JSON, it should include:

observability/
├── recording-rules/
├── alerts/
├── dashboards/
├── synthetics/
├── collectors/
├── slo/
└── tests/

When reviewing pull requests, signals, alert actions, and diagnostic entry points should all be evaluated together to avoid scenarios like "the service has been deployed, and monitoring will be added next week." CI pipelines must at minimum validate syntax, PromQL rule correctness, dashboard data source references, duplicate UID identifiers, runbook links, and potential high-cardinality dimensions.

Configuration synchronization also raises ownership concerns: if Grafana provisioning or Git sync manages dashboards, temporary edits in the UI may be overwritten or even reverse-committed into Git. Teams must clearly define which source is the authoritative one, establish procedures for urgent changes to be safely propagated back, and determine who has permission to modify shared folders and data sources.

Telemetry Retention Derived from Investigative Questions

Storing all telemetry signals permanently and at full resolution is both costly and increases data risk. Instead, retention should be stratified by use case:

• Near-real-time raw metrics: support alerting and short-term troubleshooting;
• Long-term downsampled metrics: enable capacity planning and trend analysis;
• Errors and high-latency traces: higher retention value due to diagnostic importance;
• Normal traces: sampled according to head/tail policies;
• Debug logs: short retention period with strict access controls;
• Audit logs: stored independently and secured against tampering, as required by compliance.

Begin by identifying the investigative questions and regulatory requirements, then determine resolution, sampling strategy, and retention duration. Retention decisions should not be based solely on storage cost, since query scanning, indexing, remote write network traffic, and engineering maintenance also contribute to overall cost.

Let the Platform Expose Cost and Quality Feedback

The team should be able to see what they're generating:

• active series and newly introduced label values;
• logs and traces ingestion bytes;
• rejected, dropped, and sampled telemetry;
• the most expensive and unused dashboard queries;
• alerts without owners, runbooks, or recent trigger history;
• collector queue backlogs, export failures, and data arrival latency.

Budget shouldn't just mean "turn off monitoring when you go over." A better sequence is to remove signals with no consumers, correct baselines, pre-aggregate, adjust sampling and retention, and then reassess scaling needs.

Complete Observability Delivery Requirements

Before a new service enters production, the following must be delivered:

1. SLI/SLOs and budget alerts aligned with user journeys;
2. Bounded, consistently named, and owner-assigned telemetry points;
3. A diagnostic path from overview through dependencies and resource metrics;
4. Synthetic monitoring for key user journeys, with RUM added when necessary;
5. Clear links from alerts to runbooks, dashboards, traces, logs, and revisions;
6. Versioned, tested configurations for rules, dashboards, collectors, and synthetic tests;
7. Monitoring for missing signals, telemetry pipeline failures, and cost anomalies.

The end goal of observability engineering is not to collect more data, but to shorten the path to correct decisions during outages, using as little and as trustworthy evidence as possible.

References

• OpenTelemetry, Observability Primer
• Grafana, Observability as Code
• Grafana, Provisioning
• Prometheus, The Zen of Prometheus