9.1 SLI, SLO, and Error Budgets: Making Reliability a Decision-Making Tool

A production platform runs on Kubernetes, Prometheus collects metrics, and Jaeger records request traces. Plenty of telemetry still does not define a reliability target. If the team cannot say which user events count as successful, it cannot tell how much reliability a release consumed.

SRE manages reliability with measurable user experience goals. An SLO defines how many in-scope events may fail its criteria during a window, informing release decisions and reliability investment.

Define an SLI as Good Events Divided by Eligible Events

This lesson uses an event-based SLI. Other indicators can measure time windows or other user outcomes.

SLI = good events / valid events

First define valid: which events are in scope. Then define good: the required status, latency, and result criteria. Policy can exclude clearly invalid input and test traffic. It should not automatically remove overload rejections, client cancellations caused by slow service, or authentication-infrastructure failures to improve the report.

For example, search availability:

valid = real user search requests conforming to the API contract, including server timeouts and overload rejections
good  = contract-valid search results within 800 ms, with data no more than 5 minutes stale

A legitimate "no matches" response can be successful. An unexpectedly empty result, incorrect cached data, or business failure cannot count as success merely because it returned HTTP 200. No eligible events or missing telemetry should show "no samples/unknown"; do not turn 0/0 into 100%.

SLOs have scope, window, and target

Over a rolling 30-day window, 99.9% of eligible settlement requests successfully complete within 2 seconds.

You must clearly define the service, user/region scope, event source, exclusions, time window, and target. Internal component metrics can help with diagnosis, but SLOs should prioritize aligning with the user boundary.

Different user journeys can have different SLOs; don't mix all site requests into a single average, letting low-value, high-traffic requests mask critical paths.

Error Budget

A goal of 99.9% means that within the window, up to 0.1% of valid events are allowed to fail the good criteria:

error budget = valid events × (1 - SLO target)

For example, 1,000,000 eligible events in the current window allow 1000 bad events. If 350 have occurred, 650 remain. As a rolling window advances, incoming and expiring good and bad events change those numbers. This is not a fixed monthly allowance that only decreases.

Error budgeting recognizes the cost of marginal reliability. A healthy budget permits controlled release risk; rapid budget consumption calls for pausing high-risk changes and restoring reliability.

Policies should specify trigger conditions and exception approvals in advance, not resort to budgetary adjustments after the fact to evade responsibility.

Burn rate measures spending speed

burn rate = observed bad-event ratio / allowed bad-event ratio

For a 99.9% target, the allowed bad-event ratio is 0.1%. Observing 1% bad events gives a burn rate of 10. With roughly stable traffic and a sustained ratio, a full 30-day budget would last about 3 days. Traffic changes, already consumed budget, and events leaving the window alter the actual remaining time; this is not an exact countdown.

Multi-window alerts commonly require both the long and short windows for a severity tier to exceed its threshold. The long window confirms sustained impact; the short one confirms it is still happening. Multiple tiers then cover fast outages and slower consumption, reducing transient noise and missed slow failures.

Page corresponds to user risks requiring immediate manual intervention; ticket corresponds to issues that can be addressed during work hours; dashboard is for exploration. We can't wake on-call personnel for every resource threshold.

SLO is not SLA

An SLA is a service commitment agreed with customers, potentially including consequences such as compensation when it is missed. An SLO is a target for engineering decisions and can also be public. An SLI is the actual measurement. With matching measurement definitions, teams often set a stricter SLO than the SLA to leave response margin.

SLO Quality Checks

• Are the metrics measured from the user boundary?
• Can "good/valid" be precisely defined by an event?
• Does the window align with the business cycle?
• Do low-volume services require synthetic requests or time-based SLIs?
• Could a data pipeline failure appear as if there's "no failure event"?
• Are the goals derived from user needs and costs, rather than integer preferences?
• Does the error budget policy actually influence release decisions?

The next lesson covers on-call work and incident response when the budget burns rapidly.

References

• Google SRE Workbook, Implementing SLOs
• Google SRE Workbook, Alerting on SLOs