6.2 gRPC Contracts, Streaming RPC, and Failure Boundaries

The Beacon Tower is preparing to enable services written in different languages to collaborate. But Ah Hua asks first: can an interface contract eliminate the uncertainty of remote calls?

gRPC is an RPC framework that typically uses Protocol Buffers to define contracts and maps each RPC call to an HTTP/2 stream. While it abstracts away some low-level details, a remote call is still not equivalent to a local function call, it can time out, may execute only partially, or the server might have successfully processed the request while the client never receives the result.

1. .proto Define data and service simultaneously

syntax = "proto3";

package atlas.weather.v1;

service Weather {
  rpc Get(GetRequest) returns (Reading);
  rpc Watch(WatchRequest) returns (stream Reading);
  rpc Upload(stream Reading) returns (UploadSummary);
  rpc Chat(stream ChatMessage) returns (stream ChatMessage);
}

message GetRequest {
  string city = 1;
}

message WatchRequest {
  string city = 1;
  uint32 count = 2;
}

message Reading {
  string city = 1;
  double celsius = 2;
  int64 observed_unix_seconds = 3;
}

message UploadSummary {
  uint32 accepted = 1;
}

message ChatMessage {
  string room = 1;
  string sender = 2;
  string text = 3;
}

There are four method shapes:

shape	request	response	common use cases
Unary	one	one	query or command
Server streaming	one	stream	watch, large result sets
Client streaming	stream	one	batch upload or aggregation
Bidirectional streaming	stream	stream	independent bidirectional event flow

Field numbers are part of the wire contract. After removing a field, do not reuse its number; use reserved to preserve the number and name. When adding a new field, consider how old clients will behave when they can't see it, what will be their default behavior? Do not assume semantic compatibility simply because Protobuf can decode the message.

2. How gRPC Uses HTTP/2

Each RPC corresponds to a single HTTP/2 stream. Multiple RPC streams can be multiplexed over the same connection, each maintaining its own HTTP/2 flow-control state. A typical request uses POST, with a path similar to:

/atlas.weather.v1.Weather/Get

Messages are transmitted using length-prefixed framing in gRPC, meaning the HTTP/2 DATA frame boundary does not align with the Protobuf message boundary. The final gRPC status is typically conveyed via trailers:

grpc-status: 0
grpc-message: ...

Thus, merely inspecting the HTTP status 200 is insufficient to determine whether an RPC succeeded. The client library combines the HTTP/2 transport state, gRPC status, and the application-level response. A proxy must support trailers, HTTP/2, and the gRPC content type; otherwise, scenarios like "HTTP connectivity established, but RPC fails" can occur.

Multiplexing eliminates the response ordering constraints that existed at the connection level in HTTP/1.1, but multiple streams still share a single TCP connection. Packet loss at the TCP transport layer can affect data delivery across that connection, so HTTP/2 multiplexing should not be described as eliminating "head-of-line blocking" entirely.

3. Deadline is an API contract, not an optional client feature

Clients must set deadlines for RPC calls. Without a deadline, a call may hang indefinitely, consuming thread resources, connection capacity, and upstream system resources.

Servers must:

• Monitor the remaining deadline;
• Stop unnecessary work upon client cancellation or deadline expiration;
• Pass or tighten deadlines when calling downstream services, reserving budget for cleanup operations;
• Not interpret deadline expiration as proof that the server failed to execute the operation.

Cancellation is a signal, not a transaction rollback. If an RPC may produce significant side effects, the API must support idempotency keys, operation status queries, or compensating actions.

4. Status code must preserve semantics

Common code boundaries:

• INVALID_ARGUMENT: The request itself is invalid and will not change due to system state shifts;
• FAILED_PRECONDITION: The request may be valid, but the current system state prohibits it;
• NOT_FOUND: The specified entity does not exist;
• ALREADY_EXISTS: The target for creation already exists;
• PERMISSION_DENIED: The caller's identity is known but lacks required permissions;
• UNAUTHENTICATED: No available authentication credentials are present;
• RESOURCE_EXHAUSTED: Quota or resource limit has been exceeded;
• UNAVAILABLE: A transient service availability issue, this may be suitable for controlled retry;
• DEADLINE_EXCEEDED: The caller's deadline has been exhausted;
• INTERNAL: An invariant has been violated or an internal failure that should not be exposed to clients.

Do not map all exceptions to UNKNOWN or INTERNAL, and do not directly embed database exception messages into grpc-message. Provide clients with stable, well-defined status codes and structured error details, while retaining sensitive diagnostic information in server logs or traces.

5. Retry Must Be Designed with Idempotency

Retry can originate from client libraries, service meshes, proxies, or applications, each layer adding to the load. Before enabling retry, answer these key questions:

1. Is it safe to reexecute the method?
2. Which status codes represent transient failures?
3. What are the per-attempt timeout and overall deadline?
4. How many retry attempts, with what backoff strategy and jitter?
5. How does the server use an idempotency key to prevent duplicate processing?

When a client doesn't receive a response, it cannot assume the server failed to commit. For operations like creating an order via RPC, establish a deduplication contract using a caller-generated stable request ID, never blindly retry upon seeing UNAVAILABLE.

6. Streaming Does Not Mean Infinite Buffer

While HTTP/2 and gRPC libraries provide flow control, the application itself must still enforce bounded memory policies. A streaming handler must account for:

• The maximum size allowed for a single message;
• Limits on inbound and outbound message queues;
• Backpressure applied to producers when readers are slow;
• Stopping the producer upon cancellation;
• Whether partial message progress can be resumed;
• Whether messages received before the stream's final status is confirmed can be committed.

Bidirectional streaming only indicates that message streams from both sides can advance independently, it does not automatically guarantee business ordering, exactly-once delivery, or durable replay.

7. A Runnable Unary + Server Streaming Minimum Project

Install tools:

python3 -m pip install grpcio grpcio-tools

Reduce the previous .proto to the scope required for the experiment: weather.proto

syntax = "proto3";
package atlas.weather.v1;

service Weather {
  rpc Get(GetRequest) returns (Reading);
  rpc Watch(WatchRequest) returns (stream Reading);
}

message GetRequest { string city = 1; }
message WatchRequest { string city = 1; uint32 count = 2; }
message Reading { string city = 1; double celsius = 2; }

Generate Python code:

python3 -m grpc_tools.protoc \
  -I. \
  --python_out=. \
  --grpc_python_out=. \
  weather.proto

server.py:

from concurrent import futures
import time

import grpc
import weather_pb2
import weather_pb2_grpc


class Weather(weather_pb2_grpc.WeatherServicer):
    def Get(self, request, context):
        if not request.city:
            context.abort(grpc.StatusCode.INVALID_ARGUMENT, "city is required")
        return weather_pb2.Reading(city=request.city, celsius=21.5)

    def Watch(self, request, context):
        if not request.city:
            context.abort(grpc.StatusCode.INVALID_ARGUMENT, "city is required")
        count = min(request.count or 3, 10)
        for index in range(count):
            if not context.is_active():
                return
            yield weather_pb2.Reading(
                city=request.city,
                celsius=21.5 + index * 0.1,
            )
            time.sleep(0.1)


def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
    weather_pb2_grpc.add_WeatherServicer_to_server(Weather(), server)
    if server.add_insecure_port("127.0.0.1:50051") == 0:
        raise RuntimeError("failed to bind 127.0.0.1:50051")
    server.start()
    print("listening on 127.0.0.1:50051")
    server.wait_for_termination()


if __name__ == "__main__":
    serve()

client.py:

import grpc
import weather_pb2
import weather_pb2_grpc


with grpc.insecure_channel("127.0.0.1:50051") as channel:
    stub = weather_pb2_grpc.WeatherStub(channel)
    current = stub.Get(weather_pb2.GetRequest(city="Nanjing"), timeout=2)
    print(current)

    stream = stub.Watch(
        weather_pb2.WatchRequest(city="Nanjing", count=3),
        timeout=2,
    )
    for reading in stream:
        print(reading)

The insecure channel is bound only to loopback for experimental purposes. In production, TLS server credentials or channel credentials should be used according to the trust model, and authentication and authorization must be designed at the application layer. "Enabling TLS" does not imply that the caller already has method permissions.

8. Load Balancing and Browser Boundary

gRPC channels often reuse connections over long durations. If a load balancer selects backends only at the time of TCP connection establishment, multiple RPCs over a single long-lived connection will not be re-routed request-by-request. Possible strategies include gRPC-aware L7 proxies, client-side load balancing, and xDS, all of which must be designed in concert with health checking, connection age, and outlier detection.

Browsers do not expose raw HTTP/2 framing APIs directly, and gRPC-Web typically communicates with gRPC backends via compatible clients and proxies. Its support for streaming, content type, and transport behavior differs from native gRPC, and native bidirectional streaming capabilities should not be assumed in browser clients.

9. Acceptance Issues

1. Why is an HTTP status 200 insufficient to prove a gRPC call succeeded?
2. Distinguish between deadline exceeded, cancellation, and transaction rollback.
3. Design a safe, retryable create operation and identify where the idempotency key is stored.
4. Define message limits, queue limits, and a slow-reader policy for server-streaming RPCs.
5. Why can HTTP/2 multiplexing not eliminate the connection-level impact of TCP packet loss?

References

• gRPC over HTTP/2 protocol
• gRPC Core concepts
• gRPC Deadlines
• gRPC Cancellation
• gRPC Retry
• Protocol Buffers language guide