7.3 Load Balancing Algorithms, Health Checks, and Retry Boundaries
After requests flood into the load balancer's entry point, individual backend instances quickly accumulate long queues. The on-duty operators are now asking you to design rules for request distribution and failure isolation.
A load balancer selects the destination for a request or connection among a group of backends. Its goal is not to distribute traffic so that every machine processes exactly the same number of requests per second. Instead, it aims to produce predictable service behavior under constraints of capacity, latency, availability, affinity, and cost.
1. The Information Seen at L4 and L7 Is Different
| Dimension | L4 | L7 |
|---|---|---|
| Primary Observations | Source/destination address, port, transport connection | HTTP method, host, path, fields, gRPC method, etc. |
| Common Forwarding Units | Connection / packet flow | Request / RPC / stream, depending on protocol |
| TLS Handling | Can pass through unencrypted | Requires termination when routing based on encrypted application data |
| Capabilities | NAT/DSR, connection distribution | Content routing, header policy, authentication integration, request retry |
| Cost and Risk | Less application parsing required | Higher CPU/memory usage, protocol state management, and configuration complexity |
L4 and L7 are not absolute categories in product design. A production path can simultaneously include both an L4 frontend and an L7 proxy. When TLS passthrough is used, the L7 proxy cannot see encrypted HTTP paths. Furthermore, using SNI for limited routing does not equate to full HTTP semantics.
2. Scheduling algorithm Must Match Workload
Round robin / weighted round robin
Implementation is simple and suitable when backend capacity and request cost are similar. It doesn’t know whether the previous batch of requests on a given backend was slow. Weight can represent capacity differences, but weight is not a real-time load sensor.
Least connections / least requests
Uses the number of active work items as a signal, offering a more reasonable distribution for workloads with significant duration differences compared to pure round robin. However, a single connection can carry multiple HTTP/2 or gRPC streams, so least connections doesn’t always reflect actual request load. Likewise, active requests do not indicate the CPU or I/O cost per individual request.
Latency-aware / EWMA
Uses a moving estimate of recent latency to distinguish slow backends. It must prevent low-traffic nodes from being unfairly favored due to insufficient sampling, and must also incorporate error rates, in-flight work, and warm-up phases into the decision.
Power of two choices
Randomly selects two candidate backends and then picks the one with lower load. It can effectively avoid worst-case hotspots in large backend pools with minimal query cost, but its effectiveness still depends on whether the load metric is meaningful.
Consistent / rendezvous hashing
Uses a stable key to ensure that the same tenant, cache key, or shard key consistently maps to the same backend, minimizing remapping during membership changes. It is an affinity or sharding tool, not a default load-minimizing algorithm. Hot keys can create hotspots, and uneven node capacity requires a weighted design.
3. The Smallest Experiment with Rendezvous Hashing
Rendezvous hashing (highest-random-weight hashing) computes a stable score for each key/node pair and selects the node with the highest score. When a node is removed, keys that were not originally assigned to it do not need to be remapped.
from collections import Counter
import hashlib
def score(key: str, node: str) -> int:
digest = hashlib.sha256(f"{key}\0{node}".encode()).digest()
return int.from_bytes(digest, "big")
def choose(key: str, nodes: list[str]) -> str:
if not nodes:
raise ValueError("nodes must not be empty")
return max(nodes, key=lambda node: score(key, node))
keys = [f"tenant-{i}" for i in range(10_000)]
before_nodes = ["a", "b", "c"]
after_nodes = ["a", "c"]
before = {key: choose(key, before_nodes) for key in keys}
after = {key: choose(key, after_nodes) for key in keys}
remapped = {key for key in keys if before[key] != after[key]}
assert all(before[key] == "b" for key in remapped)
print("before:", Counter(before.values()))
print("after: ", Counter(after.values()))
print("remapped:", len(remapped) / len(keys))The hash result does not guarantee that each node receives exactly 1/N keys, and uniform key distribution does not necessarily imply uniform traffic or cost. In production design, key popularity, node weight, replication strategy, and failure behavior must all be considered.
4. Health check is not "process is alive"
Active check
The load balancer proactively calls the health endpoint. These checks should be separated:
- liveness: determines whether the process needs to be restarted;
- readiness: indicates whether the backend can accept new traffic;
- deep dependency check: used for diagnostics and not necessarily suitable for directly influencing routing decisions.
If readiness imposes strong dependencies on every shared downstream service, fluctuations in downstream performance can cause all backends to report as "not ready," turning partial degradation into a complete outage.
Passive check / outlier detection
Anomalous backends are temporarily ejected based on real request metrics such as connection errors, HTTP status codes, and latency. This approach requires minimum sampling, an ejection cap, and a defined recovery policy, otherwise, during a global overload, the load balancer may sequentially remove backends, accelerating system collapse.
5. Startup, shutdown, and membership change
A new backend may not be fully warmed up even after passing health checks; JIT compilation, connection pooling, caching, and lazy initialization can all contribute to high early latency. Slow start gradually increases traffic weight, preventing the service from being overwhelmed by a sudden surge upon launch.
Planned removal should not immediately reset existing connections:
- Mark the backend as not ready or in draining mode;
- Stop assigning new requests or connections;
- Wait for in-flight work to complete, with a defined maximum grace period;
- Terminate or migrate long-lived streams that haven't finished;
- Proceed with process termination.
For protocols like WebSocket or gRPC, which support long-lived connections, stopping new connections no longer equates to stopping new application operations. Protocol-aware draining and explicit reconnect signals are required to ensure graceful handling.
6. Timeout, Retry, and Load Amplification
A load balancer retry can mask transient failures, but it can also turn a single request into multiple backend operations. When retries are stacked across multiple layers, the number of attempts can grow exponentially.
Retry policies must be evaluated in conjunction with the following considerations:
- Whether the operation is idempotent and whether an idempotency key is used;
- The overall deadline and per-attempt timeout;
- Which failures can be retried even when the request body or response headers have already been partially transmitted;
- Maximum attempts, backoff strategy, jitter, and retry budget;
- The presence and behavior of circuit breakers and load shedding;
- Whether the original attempt might still be actively running on the backend.
Hedged requests send a second request before the first one times out, even if the first one hasn't failed, this approach is only suitable for strictly controlled idempotent reads and requires a percentile threshold and additional load budget. It should not be treated as a general-purpose solution for tail latency.
7. Session affinity is a migration cost, not a free feature
Cookies, source IP, or consistent hashing can bind a caller to a backend, but they introduce uneven load distribution, failure reassignment, and difficulties during scale-in. Source IP behind NAT or proxies can make many users appear as if they're coming from the same client.
Prioritize placing durable session state in shared, stateful subsystems so stateless compute instances can be scheduled freely. When affinity is truly necessary, explicitly define TTL, backend failure behavior, rebalancing policies, and data consistency, don’t leave these decisions as implicit properties.
8. Acceptance Questions
- In HTTP/2 or gRPC workloads, why might the "least connections" backend selection strategy incorrectly pick a backend?
- Differentiate between liveness, readiness, and deep dependency checks.
- Design a slow start and connection draining strategy for a single deployment.
- If two layers each retry three times, how many total attempt opportunities are generated? How can a retry budget constrain this behavior?
- Which workloads are well-suited for rendezvous hashing, and which are better served by the least requests algorithm?