14.2 Deadline, Retries, Circuit Breakers, and Overload Protection

Remote calls will inevitably encounter slowness, disconnections, partial success, and unknown outcomes. The purpose of resilience mode is not to hide all failures, but to limit the duration of failures, resource consumption, and the spread of failures.

The typical correct order is: time-out first limits waiting, retry handles only minor transient failures, circuit breaker avoids continuous attacks on degraded dependencies, and isolation with load shedding protects its own resources.

From end-to-end deadline start

A single-hop timeout limits only one call; an end-to-end deadline indicates how much time remains for the entire user operation.

Assume an entry budget of 800 ms:

Entry processing: 80 ms
Downstream A budget 250 ms
Downstream B budget 300 ms
Serialization, queuing, and safety margin: 170 ms

The downstream shouldn't reacquire 800 ms at each layer, otherwise the deeper the call chain, the longer the total wait time. The remaining deadline should be propagated forward, and unnecessary work should be stopped as soon as the client cancels.

The timeout value comes from the delay distribution and business budget, not from arbitrarily picking an integer. Too short creates retry traffic; too long fills up threads, connections, and queues.

Retries must satisfy security conditions

What typically works well for automatic retry is:

• Idempotent read;
• Write with idempotent key;
• Clearly return transient errors and confirm that server-side operations have not been committed.

Don't retry non-idempotent writes just because of a "timeout." A timeout means the result is unknown, and the original operation might have already succeeded.

Exponential backoff with random jitter can be used:

delay = random(0, min(cap, base × 2^attempt))

Retry budget: Limit the proportion of additional requests relative to normal traffic. Otherwise, when dependency capacity drops, retries can turn a single failure into multiple times the traffic volume.

Retries should be placed as close as possible to the layer that best understands the operation's semantics. If the gateway, SDK, service mesh, and business code each retry three times, the worst case could result in 27 total attempts.

A circuit breaker protects against resource calls

A circuit breaker switches state based on the failure rate or slow call rate over a period of time:

CLOSED ── Threshold of failure reached ─> OPEN
  ▲                        │
──Probe recovery successful── HALF_OPEN

• CLOSED: Normal call and result statistics;
• OPEN: Fast fail, without further consuming connections and threads;
• HALF_OPEN: Allows a small number of probe requests to determine if recovery has occurred.

A circuit breaker is not a health check, and it doesn't guarantee that service degradation is handled correctly. It only decides whether "it's worth trying again right now." Parameters like window size, minimum sample count, slow-call threshold, and half-open probing volume all need to be tuned in conjunction with traffic conditions.

Compartmentalization, Concurrency Limits, and Load Reduction

If all dependencies share a single thread or connection pool, a slow dependency can bring the entire service to a halt. Isolation assigns independent resource limits to critical dependencies or request categories.

Queues aren't free capacity. When request arrival rates remain consistently higher than processing rates, unbounded queuing only delays failures into later timeouts. The system should exhaust resources before:

-Limit concurrency and queue length;

• Quickly reject low-priority requests;
• Return a recognizable overload signal;
• Preserve resources for health checks, control plane, and critical traffic.

Load shedding is more recoverable than timing out after accepting all requests.

Degradation must preserve business integrity

Effective downgrade example:

• Return "No recommendations available" when recommendations are not available;
• Read ranking cache and tag with update time;
• Non-critical notifications are queued for sending.

Dangerous downgrade example:

• Fails payment verification by default;
• Default authorization when permission service times out;
• Return "success" if the settlement result is unknown.

Downgrading is a business policy that should be jointly defined by product and domain teams, and cannot be automatically inferred by infrastructure libraries.

Combination Order and Observation

A single invocation can be conceptualized as:

Concurrent license → Circuit breaker → Single-attempt timeout → Budgeted retry → Result/fallback

The order of library decoration affects statistics and resource release and must be verified with fault testing. At least observe:

• Downstream consumption volume, failure rate, and latency;
• Timeout, retry count, and retry success rate;
• Breaker status and rejection count;
• Concurrency, connection pooling, queues, and load shedding;
• The proportion of downgrade results in total responses.

Mean latency masks the long tail; pay attention to p95/p99 and the percentage exceeding deadline.

References

• AWS Builders' Library, Timeouts, retries, and backoff with jitter
• Microsoft, Circuit Breaker pattern
• Google Cloud, Retry strategy