Software Systems Atlas

English

中文

English

中文

Theme

VOL 07 / CH 07 / LESSON 02

7.2 Kubernetes: Declaring State and the Control Loop ​

Kubernetes' core is not YAML; it's reconciliation: continuously comparing the desired state declared in the API with the actual state of the cluster and driving convergence toward that state.

Control Plane and Nodes ​

text
kubectl/client ─┐
Scheduler ─────┤
Controllers ───┼── API Server ── etcd
Kubelet ───────┘
  └── CRI runtime → containers
  • The API Server serves as the entry point for resource operations and authentication and authorization;
  • etcd stores the control plane's state;
  • The Scheduler selects nodes for unbound Pods;
  • Controllers create or delete objects to bring the system state into alignment;
  • The Kubelet ensures that Pods on the node match their declared specifications.

Schedulers, Controllers, and Kubelets observe or update objects through the API Server; Controllers do not directly issue every command to Kubelets. The Scheduler records a Pod’s node binding, and that node’s Kubelet uses CRI to have the container runtime create containers.

Controllers operate based on observed state, and operations must be idempotent and resilient to cache latency. A successful API call does not guarantee that all resources have converged.

Pods are scheduling and sharing boundaries ​

Containers in a Pod share its network namespace and can share declared volumes according to their mount configurations. They are scheduled together but can restart independently. emptyDir follows the Pod lifecycle, while persistent volumes need not be deleted with it. Put auxiliary processes in the same Pod only when their lifecycles and resource fates belong together.

Pods are replaceable instances. A StatefulSet can reuse a replaced Pod’s name and network identity, but the replacement has a new UID and may have a different IP. A familiar name cannot prove that the old process stopped or its data recovered.

Deployment Management of Stateless Replicas ​

A Deployment maintains replicas through a ReplicaSet and performs rolling updates. During releases, both old and new versions coexist, requiring API, message, and database compatibility.

maxUnavailable, maxSurge, readiness, and termination grace periods collectively determine the actual available capacity. Setting only replicas: 3 cannot prove that a failure or deployment phase still maintains three available, serving instances.

For example, 3 replicas with maxSurge: 1 and maxUnavailable: 0 need room to schedule an extra Pod. If that Pod stays Pending, the controller retains old replicas and waits. Check resource and scheduling constraints before deleting old replicas to "make room for the release." A Ready Pod may also need to satisfy minReadySeconds before it counts as Available, and terminating Pods may still consume resources.

Service Provides Stable Discovery ​

A normal ClusterIP Service associates backends through a selector and EndpointSlices, providing a stable virtual address and DNS name. Routing normally considers readiness, but settings such as publishNotReadyAddresses change that behavior. A readiness change also does not drain application work on existing connections. The Service does not guarantee application success and does not substitute for deadlines, retries, or business-level load balancing.

Services without selectors, ExternalName, or headless configurations have different discovery semantics; they should be chosen based on the actual protocol in use.

Three Types of Probes Must Not Be Mixed ​

  • startupProbe: Checks startup completion. The other two probes do not run until it succeeds; consecutive failures reaching the threshold terminate the container.
  • readinessProbe: Checks whether the application should receive traffic. Failure changes readiness; this probe itself does not restart the container.
  • livenessProbe: Checks for a state requiring recovery by restart. Reaching the failure threshold terminates the container, followed by handling under its restartPolicy.

Including transient database failures in the liveness probe may cause all replicas to restart simultaneously. Probes must be lightweight, fast, and include timeouts, never becoming high-cost business requests.

Requests, Limits, and Scheduling ​

The scheduler primarily places Pods based on requests; limits are enforced by the runtime. Setting requests too low leads to over-packing and intense competition during peak loads, while setting them too high wastes capacity or results in Pending states.

Applications first perform load testing to determine the capacity curve per replica, then configure resources and replica counts. For CPU HPA utilization targets, the corresponding CPU request is the denominator. Usage of 250m against a 500m request is 50%; changing the request to 250m makes the same load 100%. Without the required request, utilization may be undefined. Resource declarations directly affect scaling decisions.

HPA Is Not Instant Firefighting ​

Scaling involves delays in metrics collection, controller decisions, scheduling, image pulling, pod startup, and readiness checks. A sudden surge in traffic might exhaust the old pods before new ones are ready, so queue management, concurrency limits, traffic shedding, and reserved capacity remain essential.

Shrinking requires a stable window and must allow long-running tasks to complete or migrate. For queue consumers, backlog, processing rate, and age of the oldest message are typically closer to business requirements than CPU usage.

Chapter 11 continues with NetworkPolicy, StatefulSet, storage, and multi-tenant production governance.

References ​

Built with VitePress | Software Systems Atlas

Copyright 2026 Steven Yang