2.2 Deadline, Cancellation, Retries, and Flow-Based Backpressure

A correctly implemented RPC interface does not guarantee a reliable call chain. The caller must enforce timeouts, the server must stop unnecessary work, retries must respect idempotency, and streaming communication must limit the rate of data production.

Each Call Carries a Deadline

A deadline indicates the latest time the caller is willing to wait. It should propagate downward from end-to-end user budgets, not be re-established at each layer.

Entry remaining 900 ms
  → Service A processes 120 ms
  → Approximately 780 ms remaining when calling B
  → Already consumed time is subtracted when B calls C

Clients without a default deadline may indefinitely hold threads and connections. Excessively short deadlines can generate cancellation and retry traffic. The value should be determined through delay distributions, load testing, and business budgeting.

When a client's deadline expires, the server framework can signal cancellation, but business logic must still stop spawning new tasks and making external calls. Database transactions already submitted will not automatically roll back due to client cancellation.

Status is Part of the Contract

The semantics of common gRPC status codes should remain stable:

• INVALID_ARGUMENT: The parameters themselves are invalid;
• FAILED_PRECONDITION: The current system state does not permit the operation;
• ABORTED: A concurrency conflict, typically requiring retry at a higher layer;
• NOT_FOUND / ALREADY_EXISTS: A conflict in resource identity;
• UNAVAILABLE: Temporarily unavailable, potentially suitable for exponential backoff and retry;
• DEADLINE_EXCEEDED: The operation is pending with a timeout, write results may still be unknown;
• UNAUTHENTICATED and PERMISSION_DENIED: Missing identity or insufficient permissions.

Do not convert all business-level denials into INTERNAL. Also, do not let clients parse English error messages to decide whether to retry.

Retry Only Handles Retriable Semantics

Safe automatic retry typically requires:

• The operation to be inherently idempotent, or to carry a persistent idempotency key;
• Errors to explicitly indicate transience;
• Remaining deadline budget;
• Limits on retry count, backoff, and random jitter;
• No nesting of retries across multiple SDKs or proxies.

DEADLINE_EXCEEDED Does not prove the server did not execute. The client can call GetOperation(idempotency_key) to verify, or the server can persist the initial response for reuse on subsequent requests.

Long connections will still drop

Stream-based RPC requires a recovery protocol to be designed:

• Does each message carry a sequence/version number?
• From which checkpoint does the client resume transmission?
• How is duplicate message detection handled upon reconnection?
• How long are historical messages retained?
• How is authentication refreshed when tokens expire?
• How are heartbeat signals distinguished from business-level inactivity?

Maintaining sessions solely in memory without a recovery token means that network fluctuations will cause all long-running streams to restart from the beginning.

Backpressure and Flow Control

HTTP/2 and gRPC provide transport-layer flow control, but applications must still limit business queues. When producers generate messages continuously while consumers slow down, the system should:

• Limit in-flight messages and buffered bytes;
• Wait for consumption permission or discard data that can be safely dropped;
• Persist non-discardable data and resume from a checkpoint;
• Set upper bounds on individual message size and batch size;
• Rapidly reject new streams during overload.

An unbounded onNext loop can turn network backpressure into a process memory issue.

Load Balancing and Connection Lifecycle

Multiplexed RPCs over long-lived connections can cause uneven distribution when traditional four-layer load balancers are used. Clients or proxies must be aware of service discovery, connection aging, instance draining, and health status.

During deployment, instances first transition to not-ready, stop accepting new RPCs, and wait for in-flight unary and stream requests to reach a clear boundary. Direct termination would cause all clients to reconnect simultaneously, resulting in a spike in traffic.

Observe Each Attempt and Logical Call

A single logical call may involve multiple attempts. Metrics and traces should distinguish between:

• The success rate of the logical request;
• The status and latency of each individual attempt;
• Retry count and retry success;
• The origin of the deadline and remaining budget;
• The duration of the request flow, message rate, and reason for cancellation;
• Request/response byte sizes and compression costs.

References

• gRPC, Deadlines
• gRPC, Retry
• gRPC, Flow Control