9.2 On-call, Incident Response, Postmortems, and Toil

On-call is the final line of defense for production responsibility, not a way to pay off design debt with sleep. Sustainable on-call practices require actionable alerts, sufficient permissions, validated runbooks, clear escalation paths, and systemic improvements following any incident.

Page Must Be Actionable

An effective page should answer the following questions:

• Which user outcome is affected?
• What is the severity and scope of the impact?
• Which dashboard or runbook should the responder start with?
• What recent deployments or configuration changes occurred?
• Who is the current point of contact, and what is the escalation path?

Alerts without immediate action items should be downgraded to tickets or removed. Repeated noise trains on-call personnel to ignore actual incidents.

Major Events Require Clear Roles

• Incident Commander: Maintains priority setting, role assignments, and decision pacing;
• Operations Lead: Organizes technical diagnostics and mitigation actions;
• Communications Lead: Keeps users, support teams, and management informed;
• Scribe: Documents the timeline, hypotheses, actions, and outcomes.

Smaller incidents can be managed by a single person, but roles should be formally split as the impact grows. Uncontrolled influx of experts increases communication overhead and raises the risk of erroneous actions.

First Mitigate, Then Investigate the Root Cause

The primary goal during an incident is to minimize user impact: roll back changes, shift traffic, serve defined fallbacks, enforce traffic limits, scale out resources, or disable risky features. Full causal analysis can continue after stabilization, but mitigation must still fit the bottleneck. If the database is already out of connections, adding application replicas may add pools and retries, worsening the outage. First limit admission concurrency or disable optional queries, observe success rate and queue recovery, then decide whether scaling helps.

Every action must clearly document assumptions, the person responsible, the timestamp, expected indicators of success, and a defined rollback plan. Avoid concurrent modifications by multiple people to the same system, and never attempt irreversible fixes without proper state tracking.

Communicating Facts and Timing

Status updates include: impact, start time, current mitigation, next update time, and the facts known versus questions still unresolved. Avoid replacing evidence with phrases like "should be back soon."

External status can differ from internal technical details, but it must never obscure the actions users need to take. During handoffs, clearly communicate the current system state, unverified assumptions, and ongoing changes.

Blameless Does Not Mean No Responsibility

A postmortem examines why a particular action appeared reasonable given the information and system conditions at the time, what defenses were missing, and which contributing factors led to the outcome. It is not aimed at shaming individuals, nor does it evade accountability for decisions, ownership, or necessary improvements.

A proper postmortem should include:

• User and business impact;
• Timeline of detection, response, mitigation, and recovery;
• Contributing factors and failed defenses;
• Which mechanisms performed well;
• Action items with clear owners, deadlines, and verifiable end states.

Action items need acceptance criteria. Replace "avoid retry storms," for example, with "the payment-service owner will unify the total SDK/proxy attempt cap before a specified release, then verify through a downstream-timeout exercise that one logical call stays within that cap." Retain the exercise result and reproduction conditions. Closing the task does not prove the defense works.

Phrases like “strengthen training” or “be more careful in the future” are not actionable or verifiable. Instead, prioritize measures that reduce the likelihood of similar incidents, shrink the blast radius, or shorten recovery times.

Toil is the burden of manual work that can be automated

Toil often exhibits characteristics such as manual effort, repetitiveness, automation potential, tactical nature, linear growth, and low long-term value. Not all operations management tasks qualify as toil: designing capacity models, improving recovery protocols, and conducting drills can have significant engineering value.

Before automating, ask whether the work still has value, then simplify and standardize it. A daily manually generated report nobody reads may deserve retirement rather than a machine that produces it more diligently. Scripting unstable handbooks directly only accelerates the occurrence of errors. High-risk automation requires dry runs, idempotency, auditing, rate limiting, and rollback capabilities.

On-Call Health Metrics

• Pages per shift, night pages, and recurring alerts;
• Time to acknowledgment, mitigation, and recovery;
• Runbook hit rate and failed steps;
• Escalations and missing permissions;
• Postmortem action items completed on schedule;
• Toil time percentage and actual time saved after automation;
• On-call workload, handoff patterns, and team sustainability.

References

• Google SRE Workbook, On-Call
• Google SRE Workbook, Incident Response
• Google SRE Workbook, Postmortem Culture
• Google SRE Workbook, Eliminating Toil