7.3 Load Balancing Algorithms, Health Checks, and Retry Boundaries
A load balancer selects the destination for a request or connection among a group of backends. Its goal is not to distribute traffic so that every machine processes exactly the same number of requests per second. Instead, it aims to produce predictable service behavior under constraints of capacity, latency, availability, affinity, and cost.
1. The Information Seen at L4 and L7 Is Different
| Dimension | L4 | L7 |
|---|---|---|
| Primary Observations | Source/destination address, port, transport connection | HTTP method, host, path, fields, gRPC method, etc. |
| Common Forwarding Units | Connection / packet flow | Request / RPC / stream, depending on protocol |
| TLS Handling | Can pass encrypted traffic without terminating TLS | Requires termination when routing based on encrypted application data |
| Capabilities | NAT/DSR, connection distribution | Content routing, header policy, authentication integration, request retry |
| Cost and Risk | Less application parsing required | Higher CPU/memory usage, protocol state management, and configuration complexity |
L4 and L7 are not absolute categories in product design. A production path can simultaneously include both an L4 frontend and an L7 proxy. When TLS passthrough is used, the L7 proxy cannot see encrypted HTTP paths. Furthermore, using visible SNI for limited routing does not equate to full HTTP semantics.
2. Match Scheduling to the Workload
Round robin / weighted round robin
Implementation is simple and suitable when backend capacity and request cost are similar. It doesn’t know whether the previous batch of requests on a given backend was slow. Weight can represent capacity differences, but weight is not a real-time load sensor.
Least connections / least requests
Uses the number of active work items as a signal, offering a more reasonable distribution for workloads with significant duration differences compared to pure round robin. However, a single connection can carry multiple HTTP/2 or gRPC streams, so least connections doesn’t always reflect actual request load. Likewise, active requests do not indicate the CPU or I/O cost per individual request.
Suppose A has one HTTP/2 connection carrying 100 active RPCs, while B has eight connections with one active RPC each. Least connections picks A; least requests picks B. Even that request count needs context: B’s eight RPCs might perform expensive computation.
Latency-aware / EWMA
Uses a moving estimate of recent latency to distinguish slow backends. It must prevent low-traffic nodes from being unfairly favored due to insufficient sampling, and must also incorporate error rates, in-flight work, and warm-up phases into the decision.
Power of two choices
Randomly selects two candidate backends and then picks the one with lower load. It can effectively avoid worst-case hotspots in large backend pools with minimal query cost, but its effectiveness still depends on whether the load metric is meaningful.
Consistent / rendezvous hashing
Uses a stable key to ensure that the same tenant, cache key, or shard key consistently maps to the same backend, minimizing remapping during membership changes. It is an affinity or sharding tool, not a default load-minimizing algorithm. Hot keys can create hotspots, and uneven node capacity requires a weighted design.
3. A Rendezvous Hashing Experiment
Rendezvous hashing (highest-random-weight hashing) computes a stable score for each key/node pair and selects the node with the highest score. When a node is removed, keys that were not originally assigned to it do not need to be remapped.
from collections import Counter
import hashlib
def score(key: str, node: str) -> int:
key_bytes = key.encode("utf-8")
data = len(key_bytes).to_bytes(8, "big") + key_bytes + node.encode("utf-8")
digest = hashlib.sha256(data).digest()
return int.from_bytes(digest, "big")
def choose(key: str, nodes: list[str]) -> str:
if not nodes:
raise ValueError("nodes must not be empty")
if len(set(nodes)) != len(nodes):
raise ValueError("node identities must be unique")
return max(nodes, key=lambda node: (score(key, node), node))
keys = [f"tenant-{i}" for i in range(10_000)]
before_nodes = ["a", "b", "c"]
after_nodes = ["a", "c"]
before = {key: choose(key, before_nodes) for key in keys}
after = {key: choose(key, after_nodes) for key in keys}
remapped = {key for key in keys if before[key] != after[key]}
assert remapped == {key for key in keys if before[key] == "b"}
assert all(choose(key, list(reversed(before_nodes))) == before[key] for key in keys)
added = {key: choose(key, before_nodes + ["d"]) for key in keys}
assert all(added[key] == "d" for key in keys if added[key] != before[key])
print("before:", Counter(before.values()))
print("after: ", Counter(after.values()))
print("remapped:", len(remapped) / len(keys))The hash result does not guarantee that each node receives exactly 1/N keys, and uniform key distribution does not necessarily imply uniform traffic or cost. In production design, key popularity, node weight, replication strategy, and failure behavior must all be considered.
A length prefix prevents ambiguous concatenation when keys or node names contain delimiters. Node names break rare score ties consistently, so reordering the same membership does not change mapping. Selection hashes every node and costs linear time in the node count. It does not replicate or migrate data.
4. Health check is not "process is alive"
Active check
The load balancer proactively calls the health endpoint. These checks should be separated:
- liveness: determines whether the process needs to be restarted;
- readiness: indicates whether the backend can accept new traffic;
- deep dependency check: used for diagnostics and not necessarily suitable for directly influencing routing decisions.
If readiness imposes strong dependencies on every shared downstream service, fluctuations in downstream performance can cause all backends to report as "not ready," turning partial degradation into a complete outage.
Liveness is normally consumed by a process manager or orchestrator for restart decisions; a load balancer mainly needs readiness for new work. Intervals, thresholds, and propagation introduce detection delay, and a successful check cannot guarantee the next request succeeds. Define whether all-unhealthy behavior returns errors, sheds load, or uses a controlled fallback instead of inheriting an unknown default.
Passive check / outlier detection
Anomalous backends are temporarily ejected based on real request metrics such as connection errors, HTTP status codes, and latency. This approach requires minimum sampling, an ejection cap, and a defined recovery policy. Otherwise, during a global overload, the load balancer may sequentially remove backends, accelerating system collapse.
5. Startup, shutdown, and membership change
A new backend may not be fully warmed up even after passing health checks; JIT compilation, connection pooling, caching, and lazy initialization can all contribute to high early latency. Slow start gradually increases traffic weight, preventing the service from being overwhelmed by a sudden surge upon launch.
Planned removal should not immediately reset existing connections:
- Mark the backend as not ready or in draining mode;
- Stop assigning new requests or connections;
- Wait for in-flight work to complete, with a defined maximum grace period;
- End unfinished long-lived streams according to an application reconnect or resume contract;
- Proceed with process termination.
For protocols like WebSocket or gRPC, which support long-lived connections, stopping new connections no longer equates to stopping new application operations. Protocol-aware draining and explicit reconnect signals are required to ensure graceful handling.
HTTP/2 can use GOAWAY to stop accepting new streams while allowing existing work to finish. WebSocket needs closing signals and an application recovery mechanism. Ordinary load-balancer backend reselection does not automatically transfer existing TCP connection state, and reconnecting does not automatically replay missed business messages.
6. Timeout, Retry, and Load Amplification
A load balancer retry can mask transient failures, but it can also turn a single request into multiple backend operations. When retries are stacked across multiple layers, the number of attempts can multiply across layers.
Distinguish three retries from at most three attempts. Three retries mean four attempts including the original, so two fully exhausted layers can produce 4 × 4 = 16 downstream attempts. Three total attempts per layer give 3 × 3 = 9. These bounds ignore deadlines, backoff, and cancellation. If 100 original requests permit only 20 extra attempts, share or allocate that budget across layers instead of adding 20% independently at every layer.
Retry policies must be evaluated in conjunction with the following considerations:
- Whether the operation is idempotent and whether an idempotency key is used;
- The overall deadline and per-attempt timeout;
- Which failures can be retried even when the request body or response headers have already been partially transmitted;
- Maximum attempts, backoff strategy, jitter, and retry budget;
- The presence and behavior of circuit breakers and load shedding;
- Whether the original attempt might still be actively running on the backend.
Hedging launches an extra attempt before the first has failed. It requires safe concurrent duplication, a trigger threshold, and an additional-load limit. Reads without side effects are a common starting point; writes with idempotency keys require server-side coordination of concurrent duplicates. Cancelling a losing attempt neither proves it has stopped nor undoes its side effects.
7. Session affinity is a migration cost, not a free feature
Cookies, source IP, or consistent hashing can bind a caller to a backend, but they introduce uneven load distribution, failure reassignment, and difficulties during scale-in. Source IP behind NAT or proxies can make many users appear as if they're coming from the same client.
Prioritize placing durable session state in shared, stateful subsystems so stateless compute instances can be scheduled freely. When affinity is truly necessary, explicitly define TTL, backend failure behavior, rebalancing policies, and data consistency. Do not leave these decisions as implicit properties.
8. Acceptance Questions
- In HTTP/2 or gRPC workloads, why might the "least connections" backend selection strategy incorrectly pick a backend?
- Differentiate between liveness, readiness, and deep dependency checks.
- Design a slow start and connection draining strategy for a single deployment.
- Compare two layers with three retries each against three total attempts each. How many downstream attempts can each produce, and how can a shared retry budget constrain them?
- Which workloads are well-suited for rendezvous hashing, and which are better served by the least requests algorithm?