9.2 On-call, Incident Response, Postmortems, and Toil

On-call is the final line of defense for production responsibility, not a way to pay off design debt with sleep. Sustainable on-call practices require actionable alerts, sufficient permissions, validated runbooks, clear escalation paths, and systemic improvements following any incident.

Page Must Be Actionable

An effective page should answer the following questions:

• Which user impact is currently being affected?
• What is the severity and scope of the impact?
• Where does the issue originate, specifically, which dashboard or runbook?
• What recent deployments or configuration changes occurred?
• Who is the current point of contact, and what is the escalation path?

Alerts without immediate action items should be downgraded to tickets or removed. Repeated noise trains on-call personnel to ignore actual incidents.

Major Events Require Clear Roles

• Incident Commander: Maintains priority setting, role assignments, and decision pacing;
• Operations Lead: Organizes technical diagnostics and mitigation actions;
• Communications Lead: Keeps users, support teams, and management informed;
• Scribe: Documents the timeline, hypotheses, actions, and outcomes.

Smaller incidents can be managed by a single person, but roles should be formally split as the impact grows. Uncontrolled influx of experts increases communication overhead and raises the risk of erroneous actions.

First Mitigate, Then Investigate the Root Cause

The primary goal during an incident is to minimize user impact: roll back changes, shift traffic, downgrade services, enforce traffic limits, scale out resources, or disable risky features. Once stability is restored, deeper root cause analysis can proceed.

Every action must clearly document assumptions, the person responsible, the timestamp, expected indicators of success, and a defined rollback plan. Avoid concurrent modifications by multiple people to the same system, and never attempt irreversible fixes without proper state tracking.

Communicating Facts and Timing

Status updates include: impact, start time, current mitigation, next update time, and whether the issue is known or unknown. Avoid replacing evidence with phrases like "should be back soon."

External status can differ from internal technical details, but it must never obscure the actions users need to take. During handoffs, clearly communicate the current system state, unverified assumptions, and ongoing changes.

Blameless Does Not Mean No Responsibility

Root cause analysis focuses on understanding why a particular action appeared reasonable given the information and system conditions at the time, what defenses were missing, and what contributed factors led to the outcome. It is not aimed at shaming individuals, nor does it evade accountability for decisions, ownership, or necessary improvements.

A proper postmortem should include:

• User and business impact;
• Timeline of detection, response, mitigation, and recovery;
• Contributing factors and failure safeguards;
• Which mechanisms performed well;
• Action items with clear owners, deadlines, and verifiable end states.

Phrases like “strengthen training” or “be more careful in the future” are not actionable or verifiable. Instead, prioritize measures that reduce the likelihood of similar incidents, shrink the blast radius, or shorten recovery times.

Toil is the burden of manual work that can be automated

Toil often exhibits characteristics such as manual effort, repetitiveness, automation potential, tactical nature, linear growth, and low long-term value. Not all operations management tasks qualify as toil: designing capacity models, improving recovery protocols, and conducting drills can have significant engineering value.

Before automating, simplify and standardize the workflow. Scripting unstable handbooks directly only accelerates the occurrence of errors. High-risk automation requires dry runs, idempotency, auditing, rate limiting, and rollback capabilities.

On-Call Health Metrics

• Pages per shift, night pages, and recurring alerts;
• First response, resolution, and recovery times;
• Runbook hit rate and failed steps;
• Number of upgrades and missing permissions;
• On-call action items completed on schedule;
• Toil time percentage and actual time saved after automation;
• On-call workload, handoff patterns, and team sustainability.

References

• Google SRE Workbook, On-Call
• Google SRE Workbook, Incident Response
• Google SRE Workbook, Postmortem Culture
• Google SRE Workbook, Eliminating Toil