6.2 gRPC Contracts, Streaming RPC, and Failure Boundaries

gRPC is a remote procedure call framework, commonly using Protocol Buffers for contracts and one HTTP/2 stream per call attempt. Libraries handle wire details, but failures remain visible: a call can time out, complete partially, or succeed on the server while its result never reaches the client.

1. .proto Defines Data and Services

syntax = "proto3";

package atlas.weather.v1;

service Weather {
  rpc Get(GetRequest) returns (Reading);
  rpc Watch(WatchRequest) returns (stream Reading);
  rpc Upload(stream Reading) returns (UploadSummary);
  rpc Chat(stream ChatMessage) returns (stream ChatMessage);
}

message GetRequest {
  string city = 1;
}

message WatchRequest {
  string city = 1;
  uint32 count = 2;
}

message Reading {
  string city = 1;
  double celsius = 2;
  int64 observed_unix_seconds = 3;
}

message UploadSummary {
  uint32 accepted = 1;
}

message ChatMessage {
  string room = 1;
  string sender = 2;
  string text = 3;
}

There are four method shapes:

shape	request	response	common use cases
Unary	one	one	query or command
Server streaming	one	stream	watch, large result sets
Client streaming	stream	one	batch upload or aggregation
Bidirectional streaming	stream	stream	independent bidirectional event flow

Field numbers belong to the binary contract. Reserve removed numbers and names so old data is not misinterpreted. For new fields, check both old clients ignoring them and new servers receiving defaults. Successful decoding proves only part of compatibility. Ordinary proto3 scalars use defaults when unset; use presence-aware designs such as optional when absent and explicitly zero must differ.

2. How gRPC Uses HTTP/2

For native gRPC over HTTP/2, each call attempt occupies a stream; a retry opens another. Streams share a connection and its flow-control window while retaining individual stream windows. Requests use POST, with a path such as:

/atlas.weather.v1.Weather/Get

Each gRPC message begins with a one-byte compression flag and a four-byte big-endian length of the following message bytes, after compression when enabled. HTTP/2 DATA frames can split or combine those bytes; their boundaries do not define Protobuf messages. Final status is carried in trailers, including a permitted trailers-only response with no message body:

grpc-status: 0
grpc-message: ...

HTTP status 200 alone does not establish RPC success. The client library combines the HTTP/2 transport state, gRPC status, and the application-level response. A proxy must support trailers, HTTP/2, and the gRPC content type; otherwise, scenarios like "HTTP connectivity established, but RPC fails" can occur.

For example, uncompressed bytes 08 01 become 00 00 00 00 02 08 01 after the gRPC prefix. This prefix is five bytes, distinct from the nine-byte HTTP/2 frame header. HTTP 200 with grpc-status: 3 means INVALID_ARGUMENT; success requires final gRPC status 0.

Multiplexing eliminates the response ordering constraints that existed at the connection level in HTTP/1.1, but multiple streams still share a single TCP connection. Packet loss at the TCP transport layer can affect data delivery across that connection, so HTTP/2 multiplexing should not be described as eliminating "head-of-line blocking" entirely.

3. Deadline is an API contract, not an optional client feature

Clients must set deadlines for RPC calls. Without a deadline, a call may hang indefinitely, consuming thread resources, connection capacity, and upstream system resources.

Servers must:

• Monitor the remaining deadline;
• Stop unnecessary work upon client cancellation or deadline expiration;
• Pass or tighten deadlines when calling downstream services, reserving budget for cleanup operations;
• Not interpret deadline expiration as proof that the server failed to execute the operation.

Cancellation is a signal, not a transaction rollback. If an RPC may produce significant side effects, the API must support idempotency keys, operation status queries, or compensating actions.

A synchronous Python handler must check cancellation or use context.add_callback to stop its own work; the library cannot interrupt arbitrary computation. Read context.time_remaining() when budgeting downstream calls, reserve cleanup time, and skip the call if the budget is exhausted. Automatic deadline propagation depends on the language implementation.

4. Status code must preserve semantics

Common code boundaries:

• INVALID_ARGUMENT: The request itself is invalid and will not change due to system state shifts;
• FAILED_PRECONDITION: The request may be valid, but the current system state prohibits it;
• NOT_FOUND: The specified entity does not exist;
• ALREADY_EXISTS: The target for creation already exists;
• PERMISSION_DENIED: The caller's identity is known but lacks required permissions;
• UNAUTHENTICATED: Credentials are missing or cannot establish a valid authenticated identity;
• RESOURCE_EXHAUSTED: Quota or resource limit has been exceeded;
• UNAVAILABLE: A transient service availability issue that may permit controlled retry;
• DEADLINE_EXCEEDED: The caller's deadline has been exhausted;
• INTERNAL: A serious internal invariant violation, not a generic code for concealing every exception.

Do not map all exceptions to UNKNOWN or INTERNAL, and do not directly embed database exception messages into grpc-message. Provide clients with stable, well-defined status codes and structured error details, while retaining sensitive diagnostic information in server logs or traces.

5. Retry Must Be Designed with Idempotency

Retry can originate from client libraries, service meshes, proxies, or applications, each layer adding to the load. Before enabling retry, answer these key questions:

1. Is it safe to reexecute the method?
2. Which status codes represent transient failures?
3. What are the per-attempt timeout and overall deadline?
4. How many retry attempts, with what backoff strategy and jitter?
5. How does the server use an idempotency key to prevent duplicate processing?

When a client doesn't receive a response, it cannot assume the server failed to commit. For operations like creating an order via RPC, establish a deduplication contract using a caller-generated stable request ID. Never blindly retry upon seeing UNAVAILABLE.

gRPC can transparently retry failures known not to have reached the server application, but it has no general default status-code retry policy. Once response headers arrive, the library considers the call committed and stops retrying through this mechanism; this is separate from database commit. After partial streaming results, the application must define resumption instead of expecting retries to splice streams together.

6. Streaming Does Not Mean Infinite Buffer

While HTTP/2 and gRPC libraries provide flow control, the application itself must still enforce bounded memory policies. A streaming handler must account for:

• The maximum size allowed for a single message;
• Limits on inbound and outbound message queues;
• Backpressure applied to producers when readers are slow;
• Stopping the producer upon cancellation;
• Whether partial message progress can be resumed;
• Whether messages received before the stream's final status is confirmed can be committed.

Bidirectional streaming lets the two directions advance independently. The library preserves message order within each direction of one call, but does not provide business processing order, exactly-once delivery, or durable replay.

7. Run a Unary and Server-Streaming Project

This example is verified with Python 3.12 and the pinned dependencies below. Create an isolated environment in an empty experiment directory:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install grpcio==1.83.1 grpcio-tools==1.83.1 protobuf==7.36.1

Reduce the previous .proto to the scope required for the experiment: weather.proto

syntax = "proto3";
package atlas.weather.v1;

service Weather {
  rpc Get(GetRequest) returns (Reading);
  rpc Watch(WatchRequest) returns (stream Reading);
}

message GetRequest { string city = 1; }
message WatchRequest { string city = 1; uint32 count = 2; }
message Reading { string city = 1; double celsius = 2; }

Generate Python code:

python3 -m grpc_tools.protoc \
  -I. \
  --python_out=. \
  --grpc_python_out=. \
  weather.proto

server.py:

from concurrent import futures
import threading

import grpc
import weather_pb2
import weather_pb2_grpc

LIMITS = [("grpc.max_receive_message_length", 65536),
          ("grpc.max_send_message_length", 65536)]

class Weather(weather_pb2_grpc.WeatherServicer):
    def validate_city(self, request, context):
        if not request.city.strip() or len(request.city) > 100:
            context.abort(grpc.StatusCode.INVALID_ARGUMENT, "city needs 1-100 characters")

    def Get(self, request, context):
        self.validate_city(request, context)
        return weather_pb2.Reading(city=request.city, celsius=21.5)

    def Watch(self, request, context):
        self.validate_city(request, context)
        count = request.count or 3
        if count > 10:
            context.abort(grpc.StatusCode.INVALID_ARGUMENT, "count must be 0-10")
        stopped = threading.Event()
        if not context.add_callback(stopped.set):
            return
        for index in range(count):
            if stopped.is_set() or not context.is_active():
                return
            yield weather_pb2.Reading(city=request.city, celsius=21.5 + index * 0.1)
            if index + 1 < count and stopped.wait(0.1):
                return

def serve():
    with futures.ThreadPoolExecutor(max_workers=8) as workers:
        server = grpc.server(workers, maximum_concurrent_rpcs=16, options=LIMITS)
        weather_pb2_grpc.add_WeatherServicer_to_server(Weather(), server)
        if server.add_insecure_port("127.0.0.1:50051") == 0:
            raise RuntimeError("failed to bind 127.0.0.1:50051")
        server.start()
        print("listening on 127.0.0.1:50051", flush=True)
        try:
            server.wait_for_termination()
        except KeyboardInterrupt:
            pass
        finally:
            server.stop(grace=1).wait()

if __name__ == "__main__":
    serve()

client.py:

The client calls cancel() when leaving the stream, including early exits; it is harmless after normal completion. Each RPC has its own two-second budget, not one shared budget for the entire client.

import grpc
import weather_pb2
import weather_pb2_grpc

options = [("grpc.max_receive_message_length", 65536),
           ("grpc.max_send_message_length", 65536)]
with grpc.insecure_channel("127.0.0.1:50051", options=options) as channel:
    stub = weather_pb2_grpc.WeatherStub(channel)
    try:
        current = stub.Get(weather_pb2.GetRequest(city="Nanjing"), timeout=2)
        print(f"Get: {current.city} {current.celsius:.1f}")
        stream = stub.Watch(weather_pb2.WatchRequest(city="Nanjing", count=3), timeout=2)
        try:
            for reading in stream:
                print(f"Watch: {reading.city} {reading.celsius:.1f}")
        finally:
            stream.cancel()
    except grpc.RpcError as error:
        print(f"RPC failed: {error.code().name}")
        raise SystemExit(1)
# Get: Nanjing 21.5
# Watch: Nanjing 21.5
# Watch: Nanjing 21.6
# Watch: Nanjing 21.7

The server listens only on loopback and the client connects there, using plaintext solely for this local experiment. In two terminals with the same environment activated, run python3 server.py and python3 client.py. Output contains one current reading and three streaming readings, all fixed simulated data. The server admits at most 16 concurrent RPCs for eight worker threads. Send and receive message limits are 64 KiB each, city names are at most 100 characters, count=0 requests the default three readings, and counts above ten return INVALID_ARGUMENT. These bounds limit this example’s application load, not total process memory.

The cancellation callback wakes the wait between readings, and the loop also checks activity. Cancellation can race with the check and yield, so it does not guarantee that no further message is produced after the exact instant of cancellation. Production also needs TLS, authentication, authorization, and observability. TLS protects transport and verifies configured peer identities; it does not grant method permissions.

8. Load Balancing and Browser Boundary

gRPC channels often reuse connections over long durations. If a load balancer selects backends only at the time of TCP connection establishment, multiple RPCs over a single long-lived connection will not be re-routed request-by-request. Options include gRPC-aware L7 proxies and client-side load balancing, with xDS for configuration distribution. Combine these with health checking, connection age, and outlier detection.

Browsers do not expose raw HTTP/2 framing APIs directly, and gRPC-Web typically communicates with gRPC backends via compatible clients and proxies. Its support for streaming, content type, and transport behavior differs from native gRPC, and native bidirectional streaming capabilities should not be assumed in browser clients.

9. Acceptance Questions

1. Why is an HTTP status 200 insufficient to prove a gRPC call succeeded?
2. Distinguish between deadline exceeded, cancellation, and transaction rollback.
3. Design a safe, retryable create operation and identify where the idempotency key is stored.
4. Define message limits, queue limits, and a slow-reader policy for server-streaming RPCs.
5. Why can HTTP/2 multiplexing not eliminate the connection-level impact of TCP packet loss?

References

• gRPC over HTTP/2 protocol

• gRPC Core concepts

• gRPC Deadlines

• gRPC Cancellation

• gRPC Retry

• Protocol Buffers language guide

• gRPC Python quick start

• gRPC status codes