14.2 Deadline, Retries, Circuit Breakers, and Overload Protection

Remote calls will inevitably encounter slowness, disconnections, partial success, and unknown outcomes. Resilience patterns limit how long failures last, how many resources they consume, and how far they spread.

Start by setting the deadline for the entire operation, then decide which failures deserve retries and which resources circuit breakers, bulkheads, and load shedding protect. This is an order for reasoning; there is no universal ordering of decorators.

Start with the End-to-End Deadline

A single-hop timeout limits only one call; an end-to-end deadline indicates how much time remains for the entire user operation.

Assume A and B are called sequentially, with an entry budget of 800 ms for the whole operation:

Entry processing: 80 ms
Downstream A budget 250 ms
Downstream B budget 300 ms
Serialization, queuing, and safety margin: 170 ms

These four allocations sum to 800 ms. For parallel calls to A and B, budget along the critical path. Each downstream layer must not receive a fresh 800 ms; propagate the remaining budget. A local monotonic clock can measure elapsed time, but its raw timestamp cannot be sent to another machine. The protocol must handle clock differences; for example, gRPC propagates a timeout with elapsed time deducted.

After client cancellation, stop unnecessary work where possible. Cancellation needs server cooperation and cannot undo a committed write. Check whether the client timeout covers connection establishment, DNS, TLS, and response reading. Timing one phase does not make it an end-to-end budget.

The timeout value comes from the latency distribution and business budget, not from arbitrarily picking an integer. Too short creates retry traffic; too long fills up threads, connections, and queues.

Retries Must Satisfy Safety Conditions

What typically works well for automatic retry is:

• Idempotent reads;
• Writes with idempotency keys;
• Operations that return an explicit transient error with confirmation that the server has not committed them.

Don't retry non-idempotent writes just because of a "timeout." A timeout means the result is unknown, and the original operation might have already succeeded.

Exponential backoff with random jitter can be used:

delay = random(0, min(cap, base × 2^attempt))

Here attempt = 0 means the first retry, random samples uniformly over the given interval, and base and cap use the same time unit. If the server supplies Retry-After, consider it together with the remaining deadline. Stop if the wait and another attempt no longer fit the budget. Do not treat authentication failures, invalid parameters, circuit breaker rejections, and local overload as interchangeable transient network errors.

A retry budget limits the proportion of additional requests relative to normal traffic. Otherwise, when dependency capacity drops, retries can turn a single failure into multiple times the traffic volume.

Place retries in the layer that best understands the operation. Suppose the gateway, service SDK, and business code each allow at most 3 total attempts, including the initial call. The deepest layer can receive 3 × 3 × 3 = 27 calls. If each layer instead allows 3 additional retries, the total is 4 × 4 × 4 = 64. The difference between retries and maxAttempts can be much larger than one word.

A Circuit Breaker Protects Call Resources

A circuit breaker switches state based on the failure rate or slow call rate over a period of time:

CLOSED    ── failure threshold reached ─> OPEN
OPEN      ── waiting period ends ───────> HALF_OPEN
HALF_OPEN ── probes meet success criteria ─> CLOSED
HALF_OPEN ── probes fail ───────────────> OPEN

• CLOSED: Normal call and result statistics;
• OPEN: Fail new calls quickly without starting remote requests for them; calls already in flight are not automatically cancelled;
• HALF_OPEN: Allows a small number of probe requests to determine if recovery has occurred.

A circuit breaker is not a health check, and it doesn't guarantee that service degradation is handled correctly. It only decides whether "it's worth trying again right now." Parameters like window size, minimum sample count, slow-call threshold, and half-open probing volume all need to be tuned in conjunction with traffic conditions.

Bulkheads, Concurrency Limits, and Load Shedding

If all dependencies share a single thread or connection pool, a slow dependency can bring the entire service to a halt. Bulkheads assign independent resource limits to critical dependencies or request categories.

Queues aren't free capacity. When request arrival rates remain consistently higher than processing rates, unbounded queuing only delays failures into later timeouts. For example, with a constant arrival rate of 120 requests/second and capacity of 100/second, an empty queue with 100 slots fills in just 5 seconds in this simplified model. The queue has added no throughput. Before resources are exhausted, the system should:

• Limit concurrency and queue length;
• Quickly reject low-priority requests;
• Return a recognizable overload signal;
• Preserve resources for health checks, control plane, and critical traffic.

Load shedding is more recoverable than timing out after accepting all requests.

Degradation must preserve business integrity

Useful fallbacks include:

• Return "No recommendations available" when recommendations are not available;
• Read ranking cache and tag with update time;
• Non-critical notifications are queued for sending.

Dangerous fallbacks include:

• Treating failed payment verification as approval;
• Granting access when the permission service times out;
• Return "success" if the settlement result is unknown.

Fallback behavior is a business policy that should be jointly defined by product and domain teams, and cannot be automatically inferred by infrastructure libraries.

Composition and Observability

Here is one nesting arrangement that isolates resources per attempt:

Deadline for the entire operation
  └─ Budgeted retry loop
       ├─ Each attempt: check breaker → acquire concurrency permit → timed call → release permit
       └─ Confirm retry is safe and affordable → wait for backoff
  → Result / explicit fallback

This arrangement releases the downstream concurrency permit during backoff. Placing the permit around the whole retry loop instead limits logical calls and holds a slot during waits. These arrangements control different resources. Do not count a local permit rejection as a downstream failure; if a half-open probe permit was acquired but no call could be made, release it using the library mechanism. Decoration order affects statistics and resource release and needs fault testing. Observe at least:

• Downstream call volume, failure rate, and latency;
• Timeout, retry count, and retry success rate;
• Breaker status and rejection count;
• Concurrency, connection pooling, queues, and load shedding;
• The proportion of fallback results in total responses.

Mean latency masks the long tail; pay attention to p95/p99 and the percentage exceeding deadline.

References

• AWS, Retry with backoff pattern
• AWS, Exponential Backoff and Jitter
• Microsoft, Circuit Breaker pattern
• Google Cloud, Retry strategy