12.3 Active Observability and Observability as Code

Server-side metrics tell you what’s happening inside your system, but they don’t necessarily reflect what users actually experience. Browser rendering, DNS resolution, CDN behavior, third-party scripts, carrier network conditions, and regional routing issues can all cause users to fail tasks even when server-side dashboards look healthy.

RUM Observes Real User Distribution

Real User Monitoring (RUM) collects experience signals from actual browser or client sessions, such as navigation times, key interactions, resource loading, frontend errors, and geographic or device dimensions.

The advantages of RUM are its coverage of real-world environments and long-tail scenarios, but it comes with clear limitations:

• When traffic is low or users are absent in certain regions, there aren't enough samples to draw meaningful insights;
• Ad blockers, privacy settings, and network failures can prevent telemetry from reaching the system;
• Dimensions like users, URLs, devices, and sessions can lead to high cardinality and privacy risks;
• Averages can mask issues such as poor network conditions, low-end devices, or underrepresented regions.

When designing RUM systems, use aggregated routes, coarse-grained geographic segmentation, and controlled device categorization. Minimize, sample, limit retention periods, and enforce access controls on any data that could identify individuals. Never treat query strings, email addresses, or full user IDs as convenient diagnostic tags.

Synthetic Monitoring Exercises Critical Workflows

Synthetic monitoring is executed by controlled probes that perform fixed operations at regular intervals:

DNS resolution → TLS handshake → homepage load → login → search → place order → validate result

Synthetic accounts and data must be isolated, with operations designed to be idempotent or easily cleanable, to prevent contamination of production orders and financial reports. Probes rely on their own runner, DNS resolution, and credentials, so it's essential to distinguish between "product failure" and "probe infrastructure failure". Assertions should check the business result, such as the expected order identifier and state, rather than HTTP 200 alone; a login page can return a perfectly cheerful 200. Probe locations and frequency are deliberately selected, so probe success is not directly the SLI for real user requests.

RUM, Synthetic, and Server-Side Telemetry Complement Each Other

Signal	Best at Answering	Key Limitations
Server-side telemetry	Why services are slow or failing internally	Cannot capture browser, client, or end-user experience
RUM	Which real user combinations are affected	No samples without traffic; affected by sampling
Synthetic	Whether a probe location can complete a predefined operation	Covers only selected locations, paths, and frequencies

A regional outage can first be detected by synthetic monitoring, then validated by RUM to determine the actual scope of impact, and finally pinpointed using traces, logs, and metrics to identify specific dependencies. These three telemetry sources should share stable semantic contexts (such as service, environment, and release) and be linked via trace IDs or exemplar data, rather than forcing all events into a single storage model. Links depend on propagated context and retained target data; a missing trace may have been sampled out or expired. Missing RUM reports mean insufficient coverage, not successful user requests.

Observability as Code: Managing Executable Knowledge

Version dashboard JSON together with:

observability/
├── recording-rules/
├── alerts/
├── dashboards/
├── synthetics/
├── collectors/
├── slo/
└── tests/

When reviewing pull requests, signals, alert actions, and diagnostic entry points should all be evaluated together to avoid scenarios like "the service has been deployed, and monitoring will be added next week." CI pipelines must at minimum validate syntax, PromQL rule correctness, dashboard data source references, duplicate UID identifiers, runbook links, and potential high-cardinality dimensions.

Configuration synchronization has an owner too. With traditional Grafana file provisioning, allowUiUpdates saves UI changes to the database, not to source files; a later source update can overwrite them. Git Sync can commit UI edits through its configured workflow, subject to version, permissions, and sync mode. Define the authoritative source, the path for retaining emergency changes, and who may modify shared folders and data sources.

Telemetry Retention Derived from Investigative Questions

Storing all telemetry signals permanently and at full resolution is both costly and increases data risk. Instead, retention should be stratified by use case:

• Near-real-time raw metrics: support alerting and short-term troubleshooting;
• Long-term downsampled metrics: enable capacity planning and trend analysis;
• Errors and high-latency traces: higher retention value due to diagnostic importance;
• Normal traces: sampled according to head/tail policies;
• Debug logs: short retention period with strict access controls;
• Audit logs: stored independently and secured against tampering, as required by compliance.

Downsampling must preserve query meaning: averaging interval p99 values does not yield a daily p99, and coarse results do not preserve original peaks. The error fraction in tail-sampled traces is not the error rate of all requests either. Begin with investigative questions and applicable retention requirements, then choose resolution, sampling, and retention. Retention decisions should not be based solely on storage cost, since query scanning, indexing, remote write network traffic, and engineering maintenance also contribute to overall cost.

Let the Platform Expose Cost and Quality Feedback

The team should be able to see what they're generating:

• active series and newly introduced label values;
• logs and traces ingestion bytes;
• rejected, dropped, and sampled telemetry;
• the most expensive and unused dashboard queries;
• alerts without owners, runbooks, or recent trigger history;
• collector queue backlogs, export failures, and data arrival latency.

No recent firing does not make an alert useless: rare data-loss alerts need drills to establish that they work. Budget should not just mean "turn off monitoring when you go over." A better sequence is to remove signals with no consumers, fix cardinality, pre-aggregate, adjust sampling and retention, and then reassess scaling needs.

Complete Observability Delivery Requirements

Before a new service enters production, the following must be delivered:

1. SLI/SLOs and budget alerts aligned with user journeys;
2. Bounded, consistently named, and owner-assigned telemetry points;
3. A diagnostic path from overview through dependencies and resource metrics;
4. Synthetic monitoring for key user journeys, with RUM added when necessary;
5. Clear links from alerts to runbooks, dashboards, traces, logs, and revisions;
6. Versioned, tested configurations for rules, dashboards, collectors, and synthetic tests;
7. Monitoring for missing signals, telemetry pipeline failures, and cost anomalies.

The end goal of observability engineering is not to collect more data, but to shorten the path to correct decisions during outages, using as little and as trustworthy evidence as possible.

References

• OpenTelemetry, Observability Primer

• Grafana, Observability as Code

• Grafana, Provisioning

• Prometheus, The Zen of Prometheus

• Grafana, Git Sync