2.2 Deadlines, Cancellation, Retries, and Streaming Backpressure

A correctly implemented RPC interface does not guarantee a reliable call chain. The caller must enforce timeouts, the server must stop unnecessary work, retries must respect idempotency, and streaming communication must limit the rate of data production.

Each Call Carries a Deadline

A deadline indicates the latest time the caller is willing to wait. It should propagate downward from end-to-end user budgets, not be re-established at each layer.

Entry remaining 900 ms
  → Service A processes 120 ms
  → Approximately 780 ms remaining when calling B
  → Already consumed time is subtracted when B calls C

Propagate remaining duration rather than treating absolute clocks on different machines as one shared deadline. gRPC deadline propagation subtracts elapsed time, but automatic propagation depends on the language and call path. Reserve time for returning the response and cleaning up resources too.

Clients without a default deadline may retain call-related resources indefinitely. Excessively short deadlines can generate cancellation and retry traffic. The value should be determined through delay distributions, load testing, and business budgeting.

When a client's deadline expires, the server framework can signal cancellation, but business code must cooperatively check cancellation and propagate it to child tasks and external calls. Database transactions already committed will not automatically roll back due to client cancellation.

Status is Part of the Contract

The semantics of common gRPC status codes should remain stable:

• INVALID_ARGUMENT: The parameters themselves are invalid;
• FAILED_PRECONDITION: The current system state does not permit the operation;
• ABORTED: A concurrency conflict, typically requiring retry at a higher layer;
• NOT_FOUND: The target resource does not exist; ALREADY_EXISTS: The resource being created already exists;
• UNAVAILABLE: Temporarily unavailable, potentially suitable for exponential backoff and retry;
• DEADLINE_EXCEEDED: The waiting deadline expired; a write outcome may still be unknown;
• UNAUTHENTICATED and PERMISSION_DENIED: No valid authentication credentials or insufficient permissions.

Do not convert all business-level denials into INTERNAL. Also, do not let clients parse English error messages to decide whether to retry.

Retry Only Handles Retriable Semantics

Safe automatic retry typically requires:

• An inherently idempotent operation, or server-side durable handling of repeated requests with the same idempotency key;
• Errors to explicitly indicate transience;
• Remaining deadline budget;
• Limits on retry count, backoff, and random jitter;
• One layer owns retries, avoiding multiplied attempts across SDKs and proxies.

DEADLINE_EXCEEDED does not prove the server did not execute. An application can separately provide GetOperation(idempotency_key) or persist the initial result for repeated requests. The previous lesson’s scoring query contract does not implement that operation-query API.

This gRPC service config applies only to the idempotent GetScore query. Load it through the client channel’s configuration mechanism and still set a deadline on each call. maxAttempts: 3 includes the initial attempt, allowing at most two extra attempts. All attempts and backoff share the logical call’s time budget.

{
  "methodConfig": [{
    "name": [{"service": "atlas.scoring.v1.ScoringService", "method": "GetScore"}],
    "retryPolicy": {
      "maxAttempts": 3,
      "initialBackoff": "0.1s",
      "maxBackoff": "0.4s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

Once response headers arrive, gRPC normally considers the RPC committed and stops retrying under this policy. Resuming an interrupted stream remains the application protocol’s job. Transparent framework retries may also occur without an explicit retry policy when gRPC can establish that the request has not reached server application processing. This does not make arbitrary write replay safe.

Long connections will still drop

Stream-based RPC requires a recovery protocol to be designed:

• Does each message carry a sequence/version number?
• From which checkpoint does the client resume transmission?
• How is duplicate message detection handled upon reconnection?
• How long are historical messages retained?
• How is authentication refreshed when tokens expire?
• How are heartbeat signals distinguished from business-level inactivity?

Maintaining sessions solely in memory without a recovery token means that network fluctuations will cause all long-running streams to restart from the beginning.

Backpressure and Flow Control

HTTP/2 and gRPC provide transport-layer flow control, but applications must still limit business queues. When producers generate messages continuously while consumers slow down, the system should:

• Limit in-flight messages and buffered bytes;
• Wait for consumption permission or discard data that can be safely dropped;
• Persist non-discardable data and resume from a checkpoint;
• Set upper bounds on individual message size and batch size;
• Rapidly reject new streams during overload.

An unbounded onNext loop can turn network backpressure into a process memory issue. At 64 KiB per message, a queue limited to 1000 messages can still hold about 62.5 MiB of payload before object overhead. Limit both message count and bytes. In a bidirectional stream, neither side should only write without reading: incorrectly configured manual flow control can leave both waiting for a window, with plenty of connection activity and no business progress.

Load Balancing and Connection Lifecycle

Multiplexed RPCs over long-lived connections can cause uneven distribution when traditional Layer 4 load balancers are used. Clients or proxies must be aware of service discovery, connection aging, instance draining, and health status.

During deployment, remove the instance from discovery and start server-side graceful shutdown, then let in-flight unary and streaming calls reach a defined boundary. Marking an instance not-ready does not close established HTTP/2 connections or make discovery updates instantaneous. Long streams need migration or resume support, with a drain deadline; otherwise graceful shutdown can wait indefinitely. Abrupt termination may cause many clients to reconnect together.

Observe Each Attempt and Logical Call

A single logical call may involve multiple attempts. Metrics and traces should distinguish between:

• The success rate of the logical request;
• The status and latency of each individual attempt;
• Retry count and retry success;
• The origin of the deadline and remaining budget;
• Stream duration, message rate, and reason for cancellation;
• Request/response byte sizes and compression costs.

References

• gRPC, Deadlines
• gRPC, Cancellation
• gRPC, Status Codes
• gRPC, Graceful Shutdown
• gRPC, Retry
• gRPC, Flow Control