7.2 Kubernetes: Declaring State and the Control Loop

Kubernetes' core is not YAML; it's reconciliation: continuously comparing the desired state declared in the API with the actual state of the cluster and driving convergence toward that state.

Control Plane and Nodes

kubectl/client → API Server → etcd
                    │
          ┌─────────┴─────────┐
       Scheduler          Controllers
                              │
                         Kubelet on Node

• The API Server serves as the entry point for resource operations and authentication and authorization;
• etcd stores the control plane's state;
• The Scheduler selects nodes for unbound Pods;
• Controllers create or delete objects to bring the system state into alignment;
• The Kubelet ensures that Pods on the node match their declared specifications.

Controllers operate based on observed state, and operations must be idempotent and resilient to cache latency. A successful API call does not guarantee that all resources have converged.

Pods are scheduling and sharing boundaries

Containers within a Pod share the same network namespace and volume lifecycle, and are typically scheduled and terminated together. Only auxiliary processes whose lifetimes and resource allocations are tightly bound should be placed within the same Pod.

A Pod is a replaceable instance, not a permanent server. Do not rely on Pod names, IPs, or local disks for long-term stability.

Deployment Management of Stateless Replicas

A Deployment maintains replicas through a ReplicaSet and performs rolling updates. During releases, both old and new versions coexist, requiring API, message, and database compatibility.

maxUnavailable, maxSurge, readiness, and termination grace periods collectively determine the actual available capacity. Setting only replicas: 3 cannot prove that a failure or deployment phase still maintains three available, serving instances.

Service Provides Stable Discovery

A Service associates ready backend endpoints via selector/EndpointSlice, offering callers a stable virtual address and DNS name. The Service does not guarantee application success and does not substitute for deadlines, retries, or business-level load balancing.

Services without selectors, ExternalName, or headless configurations have different discovery semantics; they should be chosen based on the actual protocol in use.

Three Types of Probes Must Not Be Mixed

• startupProbe: Determines whether the application has completed its slow startup; it can suppress other probes during the initial phase.
• readinessProbe: Indicates whether the application is ready to receive traffic; failure does not necessarily trigger a restart.
• livenessProbe: Checks whether the process has entered a state that can only be recovered by restarting.

Including transient database failures in the liveness probe may cause all replicas to restart simultaneously. Probes must be lightweight, fast, and include timeouts, never becoming high-cost business requests.

Requests, Limits, and Scheduling

The scheduler primarily places Pods based on requests; limits are enforced by the runtime. Setting requests too low leads to over-packing and intense competition during peak loads, while setting them too high wastes capacity or results in Pending states.

Applications first perform load testing to determine the capacity curve per replica, then configure resources and replica counts. When using CPU Horizontal Pod Autoscaler (HPA) with requests as the target, inaccurate requests can cause distorted scaling behavior.

HPA Is Not Instant Firefighting

Scaling involves delays in metrics collection, controller decisions, scheduling, image pulling, pod startup, and readiness checks. A sudden surge in traffic might exhaust the old pods before new ones are ready, so queue management, concurrency limits, traffic shedding, and reserved capacity remain essential.

Shrinking requires a stable window and must allow long-running tasks to complete or migrate. For queue consumers, backlog, processing rate, and age of the oldest message are typically closer to business requirements than CPU usage.

Chapter 11 continues with NetworkPolicy, StatefulSet, storage, and multi-tenant production governance.

References

• Kubernetes, Kubernetes Components
• Kubernetes, Pods
• Kubernetes, Deployments
• Kubernetes, Horizontal Pod Autoscaling