12.2 Alerts and Diagnostic Dashboards: From Signal to Action

The output of an alerting system is not a notification; it is timely, accurate human action. An expression may be mathematically correct, but if the recipient does not understand its impact, urgency, or the next step required, it remains a failed alert design.

Page on User Risk

Chapter 9 already used burn rate to assess error budget consumption. Here we add operational contracts. Every alert that wakes someone up must meet at least the following criteria:

1. A significant user impact is either occurring or about to occur;
2. Action taken now can improve the outcome;
3. The recipient is clearly identified and has the authority to act;
4. The annotation must clearly point to the dashboard, runbook, and change context;
5. Data gaps or missing values must not be mistaken for normal baselines.

A CPU usage of 80% alone typically fails to meet these conditions. If CPU saturation would cause an SLO degradation and rapidly consume the error budget, leading to user impact, page on the user impact and use CPU as a diagnostic signal. If, however, the high usage is merely a long-term capacity trend, a work item for capacity planning should be created.

for Delays Firing; keep_firing_for Delays Resolution

Assume a 99.9% request-based SLO over 30 days, with one-hour and five-minute error-ratio recording rules using the same eligible requests and service/failure-domain labels. 0.0144 / 0.001 = 14.4 times the allowed error rate; at roughly stable traffic, one hour consumes about 2% of the full-window budget. This is one fast-burn tier; slower windows need separate rules:

groups:
  - name: atlas-slo
    rules:
      - alert: AtlasApiFastBudgetBurn
        expr: |
          (service:slo_errors_per_request:ratio_rate1h{service="atlas-api"} > 0.0144)
          and
          (service:slo_errors_per_request:ratio_rate5m{service="atlas-api"} > 0.0144)
        for: 5m
        keep_firing_for: 10m
        labels:
          severity: page
          service: atlas-api
        annotations:
          summary: "atlas-api is consuming error budget rapidly"
          impact: "Eligible settlement API requests are failing or timing out"
          dashboard: "https://grafana.example/d/atlas-api"
          runbook: "https://runbooks.example/atlas-api/budget-burn"

Both windows must exceed the threshold before an alert becomes pending. The same label set must then satisfy for: 5m continuously to fire. Account for that added detection delay instead of copying it unconditionally. Prometheus creates alerts for returned vector elements: do not replace > with > bool, which also returns elements with value zero.

keep_firing_for: 10m keeps an alert firing after the condition stops matching and can also hide short data gaps. Missing data does not thereby prove recovery; monitor collection health independently. Derive thresholds, windows, and added delays together.

The rules also need unit tests: normal cases, threshold boundaries, continuous triggers, missing data, and recovery sequences must all be covered.

Alertmanager Routes Notifications

Prometheus determines what conditions are triggering alerts, while Alertmanager handles deduplication, grouping, routing, silencing, and inhibition.

This example includes receiver definitions so its configuration structure can be checked. Replace the URLs with real notification integrations and configure their required credentials. Successful parsing does not prove webhook delivery:

route:
  receiver: default-ticket
  group_by: [alertname, service, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: primary-oncall
receivers:
  - name: default-ticket
    webhook_configs:
      - url: https://notifications.example/tickets
  - name: primary-oncall
    webhook_configs:
      - url: https://notifications.example/oncall

group_wait delays a new group’s first notification, group_interval controls notifications of subsequent group changes, and repeat_interval controls reminders for unchanged alerts. These timers are distinct from the rule’s for. An overly granular group_by can result in hundreds of notifications for a single incident, while being too broad risks merging events that belong to different ownership domains. Grouping keys should preserve responsibility and failure domains, never default to including volatile dimensions like pod and instance.

Inhibition expresses the logic that when a root cause alert is active, it suppresses symptom alerts within the configured scope, such as suppressing individual service unreachabilities within the affected cluster during a network outage. Explicit source/target matchers and equal failure-domain labels constrain the scope. Inhibition does not infer causality or stop rules from firing; it suppresses notifications. Silence is a time-bound, creator- and reason-tagged manual rule, ideal for planned maintenance; it is not a long-term repository for suppressing bad alerts.

For high-availability Alertmanager deployments, it's essential to validate the end-to-end notification flow. Monitoring only process liveness of individual components is insufficient to ensure alerts propagate reliably from rules to on-call endpoints. Regular end-to-end canary notifications should be run to verify the integrity of the entire alerting chain.

Arrange Dashboards by Diagnostic Question

A service diagnostics interface can be structured into three layers:

Layer 1: Is the user impacted?
  SLOs, request volume, error rates, latency distribution

Layer 2: Where is the impact occurring?
  region, version, route, dependency, tenant tier

Layer 3: Why is it happening?
  saturation, queues, connection pools, GC, downstream latency, trace/log links

The initial screen should not overwhelm users with CPU, memory, and dozens of node curves. Instead, start with clear context: range, time period, units, target baselines, and release annotations, enabling on-call engineers to assess impact within one minute. Then allow drilling down from stable, low-cardinality dimensions.

Panel titles should answer a specific question, such as "Which regions are consuming budget in the past 30 minutes?" rather than "Error Graph 7." Percentages should consistently display as 0–1 or 0–100, and latency values (whether in seconds or milliseconds) must be explicitly labeled in both the title and axes.

Put Changes on the Same Timeline

A large number of incidents are tied to changes in deployments, configurations, scaling, and dependencies. The dashboard should overlay deployment revisions, feature flags, schema migrations, and incident annotations on a single timeline, and provide links from anomaly points to:

• The current image digest;
• The corresponding Git commit and release approval;
• A representative trace;
• Log query filters limited by service, region, and version.

This approach is far more effective at identifying root causes than adding another ten resource panels.

Alert Review Is an Ongoing Practice

After each incident, document which alerts first provided actionable signals, which were redundant, delayed, or lacked context, and which failed to trigger despite being expected. For each alert page, track its frequency, acknowledgment time, actionability, and percentage of cases that self-resolved automatically. Pages that consistently require no human action should be downgraded, merged, or removed. If action is needed but nobody responds, fix on-call routing and ownership instead of deleting the alert.

The next lesson will fill in the gaps that server-side telemetry cannot cover: real user experience, synthetic monitoring, versioned configurations, and telemetry cost control.

References

• Prometheus, Alerting practices

• Prometheus, Alerting rules

• Prometheus, Alertmanager

• Prometheus, Alertmanager configuration

• Google SRE, Alerting on SLOs

• Prometheus, Rule unit tests