14.1 Service Discovery, Gateway, BFF, and Configuration Boundaries

After services are decomposed, the caller's first challenge isn't "how to send an HTTP request"; it's that instance addresses change dynamically, instances may not be ready yet, entry points require centralized governance, and different clients need different data shapes.

This lesson establishes the call topology. Timeout, retry, circuit breaking, and overload protection will be covered in the next lesson.

Service Discovery Answers Two Questions

Service discovery must answer:

1. Which instances belong to the target service?
2. Which instances are currently ready to receive requests?

There are two common approaches:

• Client-side discovery: The caller queries a registry to obtain instance information and performs load balancing.
• Server-side discovery: The caller accesses a stable endpoint, and a load balancer or platform proxy selects the appropriate instance.

Consider a normal Kubernetes ClusterIP Service with a selector: DNS returns the Service virtual IP, and the platform uses endpoint conditions in EndpointSlices to route traffic to ready Pods. The example assumes a Service named scoring, the default namespace, and a cluster domain of cluster.local. Application code should not cache Pod IPs as permanent addresses.

caller
    ↓ DNS
scoring.default.svc.cluster.local
    ↓ Service / EndpointSlice
ready scoring Pods

A headless Service can return endpoint addresses directly through DNS, so the virtual IP path does not apply. Setting publishNotReadyAddresses: true also makes the EndpointSlice ready condition true; accessing a Service does not unconditionally mean accessing only ready instances. Readiness changes take time to propagate, and existing connections and in-flight requests still need draining. See the Service and EndpointSlice conditions documentation.

Repeated liveness probe failures trigger a container restart. A failed readiness probe removes the instance from normal traffic routing without itself triggering a restart. When a startup probe is configured, liveness and readiness checks wait for it to succeed. Treating a transient database failure as a liveness failure can restart every instance and amplify the outage.

API Gateway is a boundary, not a business center

A gateway is well-suited to handle consistency responsibilities across multiple entry points:

• TLS termination, authentication, and basic authorization;
• Routing, protocol translation, and version migration;
• Request size limits, rate limiting, and entry-point observability;
• Lightweight, stable response aggregation.

The gateway should not contain core business logic. If rules such as "calculate scores, update standings, send rewards" are implemented within the gateway, internal calls, batch processing, and message ingestion will bypass it, turning the gateway into a tightly coupled release bottleneck.

The gateway must also propagate client identity, tracing context, and cancellation signals. It must remove client-supplied internal identity headers and forward credentials that downstream services can verify. Those services still need to check whether the user may act on the specified object. Passing through the gateway does not grant an administrator pass to every tournament.

BFF Is Responsible for Client Experience

Web, mobile, and third-party APIs often have distinct requirements:

• Mobile apps prioritize fewer round trips and performance under poor network conditions;
• Web clients may require finer-grained, parallelizable data;
• Third-party APIs demand long-term compatibility and strict usage quotas.

A BFF (Backend for Frontend) can provide a stable presentation model tailored to a specific client type:

Mobile App → Mobile BFF ─┬→ Player Service
                         ├→ Ranking Service
                         └→ Reward Service

The BFF can filter and aggregate data, but it should not become another shared enterprise bus. Each additional fan-out introduces complexity that must be evaluated in terms of the slowest dependent service, partial failure risks, and overall latency budget.

Suppose three required dependencies each have a 99.9% success rate over the same window, their failures are independent, and we exclude BFF failures and additional timeouts. The probability of obtaining all three results is approximately:

0.999 × 0.999 × 0.999 ≈ 99.70%

Parallel calls do not make failures independent; a shared database outage breaks that assumption. Parallel waiting usually takes as long as the slowest dependency, but success still requires all three to succeed. If recommendations are optional, an explicit "No recommendations available" response can keep their failure from failing the whole page.

Configuration and Secrets Are Not the Same Thing

Configuration can be categorized by sensitivity and how changes are made:

Type	Example	Requirements
General deployment configuration	Ports, dependency addresses, feature parameters	Versioned, auditable, environment-specific
Dynamic runtime configuration	Rate limiting thresholds, experiment ratios	Range validation, rollback capability, and observability of activation
Secrets	Database passwords, signing keys	Dedicated secret storage, principle of least privilege, regular rotation
Business data	Product prices, competition rules	Must follow business models and approval workflows; must not be disguised as configuration

Never commit secrets to version control. And don't allow arbitrary runtime modifications to core business logic simply because dynamic configuration is convenient. Configuration changes require ownership, validation, auditing, and rollback capability.

Kubernetes Secret data uses Base64 encoding, which provides no confidentiality; encryption at rest and access permissions still need configuration. When rotating a database password, verify when connection pools establish new connections, not just whether the configuration service says "updated." Dynamic configuration does not guarantee simultaneous activation across instances either. Record configuration versions and check what each instance actually uses.

Shorter Topologies Are Easier to Diagnose

If an entry path follows this structure:

Gateway → BFF → Aggregator → Domain Service → Internal Proxy → Database

Each hop consumes latency, connection budget, and availability. Before adding any intermediate layer, clearly define the stable responsibility it serves. Introducing a proxy solely for "potential future reuse" shifts future costs into current failure surfaces.

Design Checklist

• Does the service use a stable name for discovery, or does it rely on volatile instance addresses?
• Do readiness and liveness probes express distinct semantics?
• Does the gateway perform only boundary governance, or does it contain business logic?
• Does each fan-out dependency in the BFF have a partial failure strategy?
• Are configurations and secrets stored, accessed, and rotated with separate mechanisms and permissions?
• Can we trace from the entry point to all downstream services, and propagate deadlines and cancellation signals?

Next lesson, we’ll add failure control to this topology: first limit waiting time, then limit retries, and finally constrain resource pool consumption during failures.

References

• Kubernetes, Service
• Kubernetes, Liveness, Readiness, and Startup Probes
• Kubernetes, Secrets
• Microsoft, Backends for Frontends pattern