11.3 Kubernetes Capacity Governance: Scheduling, Scaling, and Interruptions

Kubernetes can place Pods on nodes and adjust replica counts, but it doesn't understand which levels of latency, cost, or fault risk are acceptable to business operations. Effective production capacity governance treats requests, limits, scheduling constraints, HPA, PDB, and tenant boundaries as an interconnected control system.

Requests Determine Scheduling, Limits Constrain Execution

resources:
  requests:
    cpu: 500m
    memory: 768Mi
  limits:
    cpu: "2"
    memory: 1Gi

The scheduler evaluates node capacity based on requests, not on real-time resource usage. High request values can make a node appear full, while low estimates may lead to overpacking during peak loads.

CPU limits are typically enforced through throttling, which can introduce tail latency even when there's still available CPU. Memory limits, if exceeded, may result in OOM (Out-of-Memory) kills. Whether to set CPU limits should be determined by multi-tenant risk and latency testing, never apply the same configuration to all services. Memory requests and limits should be set in conjunction with workset size, peak usage, and the cost of OOM recovery.

Replicas Must Be Distributed Across Fault Domains

If three replicas are all scheduled on the same node or availability zone, the number of replicas does not translate into availability. To achieve distribution goals, topology spread constraints are used:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: atlas-api
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: atlas-api

Hard constraints enhance isolation but may cause Pod Pending if capacity is insufficient; soft constraints improve schedulability but may compromise distribution. The choice must align with the reserved capacity of each fault domain.

Taints and tolerations allow specific Pods to run on nodes, but they do not guarantee placement on a particular node; for targeted scheduling, node affinity must also be applied. Avoid using user-added labels to enforce security isolation, trusted node labels must be maintained by controlled identities.

HPA's Denominator Comes from Request

The CPU utilization target is roughly based on:

Current usage / CPU request

If a container lacks an appropriate request, the HPA cannot accurately compute its resource utilization. Changing the request (while real workload remains constant) will alter the percentage the HPA observes.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: atlas-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: atlas-api
  minReplicas: 3
  maxReplicas: 30
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

For queue consumers, backlog volume or age of the oldest message typically aligns more closely with actual demand than CPU usage. For latency-sensitive services, combine concurrency levels with business SLIs. Pipeline failures, cold start times, and node scaling delays must be included in capacity planning; otherwise, the HPA will only generate a batch of Pending Pods.

Don't let HPA and VPA compete for the same resource signal without analyzing the feedback loop: when VPA modifies a request, it changes the denominator in HPA's utilization calculation.

PDB Only Constrains Voluntary Evictions

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: atlas-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: atlas-api

PDB restricts voluntary evictions initiated through the Eviction API, such as when draining a node in accordance with PDB policies. It does not prevent node crashes, nor does it limit rolling updates performed by Deployments. PDB cannot replace the need for sufficient replicas and cross-zone distribution.

Overly restrictive PDB configurations can leave nodes permanently unmanageable. Before applying such rules, developers must simultaneously evaluate replica counts, maxUnavailable settings, readiness probes, termination grace periods, and the cluster's overall upgrade strategy.

Namespace Is Not a Complete Security Boundary

In shared clusters, namespaces are often combined with several controls:

• RBAC: defining who can operate which API objects;
• ResourceQuota: limiting the total requests, limits, and object counts within a namespace;
• LimitRange: setting default values and upper/lower bounds for individual container configurations;
• NetworkPolicy: restricting inter-container communication;
• Pod Security Admission: blocking high-risk pod configurations;
• Dedicated node pools, RuntimeClass, or isolated clusters: for scenarios requiring stronger isolation.

ResourceQuota operates independently of the actual cluster capacity. The sum of quotas across all namespaces can exceed the available node resources, meaning it cannot substitute for capacity planning. Additionally, namespaces share the control plane and portions of the node kernel. For tenants that do not trust each other or for workloads with strict compliance requirements, a stronger security boundary must be evaluated.

Production Readiness Checklist

Release: Rolling strategy, readiness probes, graceful shutdown, rollback and migration compatibility
Capacity: Requests/limits, peak utilization margin, HPA metrics, node scaling latency
Fault tolerance: Cross-node and cross-region distribution, Pod Disruption Budgets, dependency timeouts, recovery drills
Security: Workload identities, RBAC, PSA, NetworkPolicy, image signing
Storage: PVC lifecycle, snapshot consistency, RPO/RTO, offsite recovery
Operations: Event signaling, change auditing, controller health, certificate and key rotation

The value of this checklist isn't in checking off items once; it's in transforming those conditions into admission policies, CI checks, continuous monitoring, and regular drills.

References

• Kubernetes, Resource Management for Pods and Containers
• Kubernetes, Horizontal Pod Autoscaling
• Kubernetes, Disruptions
• Kubernetes, Resource Quotas
• Kubernetes, Pod Topology Spread Constraints