11.3 Kubernetes Capacity Governance: Scheduling, Scaling, and Disruptions

Kubernetes can place Pods on nodes and adjust replica counts, but it doesn't understand which levels of latency, cost, or fault risk are acceptable to business operations. Effective production capacity governance treats requests, limits, scheduling constraints, HPA, PDB, and tenant boundaries as an interconnected control system.

Requests Determine Scheduling, Limits Constrain Execution

This is the resources field for one container, to be placed inside a Pod template’s container configuration:

resources:
  requests:
    cpu: 500m
    memory: 768Mi
  limits:
    cpu: "2"
    memory: 1Gi

The scheduler evaluates node capacity based on requests, not on real-time resource usage. High request values can make a node appear full, while low estimates may lead to overpacking during peak loads.

CPU limits are typically enforced through throttling, which can introduce tail latency even when there's still available CPU. Memory limits, if exceeded, may result in OOM (Out-of-Memory) kills. Whether to set CPU limits should be determined by multi-tenant risk and latency testing. Do not apply the same configuration to all services. Memory requests and limits should be set in conjunction with working-set size, peak usage, and the cost of OOM recovery.

Replicas Must Be Distributed Across Fault Domains

If three replicas are all scheduled on the same node or availability zone, the number of replicas does not translate into availability. This Pod spec fragment assumes the Pod has label app: atlas-api, trustworthy and correct zone/hostname node labels, and three zones eligible for scheduling:

topologySpreadConstraints:
  - maxSkew: 1
    minDomains: 3
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: atlas-api
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: atlas-api

maxSkew: 1 bounds count imbalance; by itself it does not require three zones. With only one eligible zone, the default behavior can place all three replicas there. Adding minDomains: 3 treats the global minimum as zero when fewer than three domains qualify. This prevents piling more replicas into existing domains, but may leave replacement Pods Pending during a failure. The soft hostname constraint only affects scheduling preference. Choose the trade-off between distribution and capacity during failures using the headroom in each domain.

Taints and tolerations allow specific Pods to run on nodes, but they do not guarantee placement on a particular node; for targeted scheduling, node affinity must also be applied. Avoid using user-added labels to enforce security isolation; trusted node labels must be maintained by controlled identities.

HPA's Denominator Comes from Request

The CPU utilization target is roughly based on:

Current usage / CPU request

If a container lacks an appropriate request, the HPA cannot accurately compute its resource utilization. Changing the request (while real workload remains constant) will alter the percentage the HPA observes.

This example assumes an existing atlas-api Deployment in the same namespace, CPU requests on all its containers, and a working resource metrics API:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: atlas-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: atlas-api
  minReplicas: 3
  maxReplicas: 30
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Ignoring tolerance, missing metrics, and startup adjustments, the desired count is approximately ceil(current replicas × current utilization / target utilization). Three replicas at 90% with a 60% target give five replicas, then bounds and scaling policies apply. The 300-second downscale stabilization window uses the higher recommendations in that window; it is not simply a fresh 300-second sleep for every change.

For queue consumers, backlog or oldest-message age is often closer to demand than CPU, and needs a suitable custom/external metrics API. Latency-sensitive services can scale from concurrency and use business SLIs to verify the effect. If latency comes from an overloaded shared database, more application replicas may not help. Pipeline failures, cold start times, and node scaling delays must be included in capacity planning; otherwise, the HPA will only generate a batch of Pending Pods.

Don't let HPA and VPA compete for the same resource signal without analyzing the feedback loop: when VPA modifies a request, it changes the denominator in HPA's utilization calculation.

PDB Only Constrains Voluntary Evictions

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: atlas-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: atlas-api

PDB restricts voluntary evictions initiated through the Eviction API, such as when draining a node in accordance with PDB policies. It does not prevent node crashes, nor does it limit rolling updates performed by Deployments. PDB cannot replace the need for sufficient replicas and cross-zone distribution.

With three matching Ready Pods, the example permits one eviction. If one is already unhealthy, it normally permits no further eviction of healthy Pods. PDB also does not block direct Pod deletion or workload scale-down. If unhealthy Pods block a drain, evaluate unhealthyPodEvictionPolicy: AlwaysAllow after checking how they will recover.

Overly restrictive PDBs can block node maintenance. Before applying such rules, developers must simultaneously evaluate replica counts, maxUnavailable settings, readiness probes, termination grace periods, and the cluster's overall upgrade strategy.

Namespace Is Not a Complete Security Boundary

In shared clusters, namespaces are often combined with several controls:

• RBAC: defining who can operate which API objects;
• ResourceQuota: limiting the total requests, limits, and object counts within a namespace;
• LimitRange: setting default values and upper/lower bounds for individual container configurations;
• NetworkPolicy: restricting Pod communication;
• Pod Security Admission: blocking high-risk pod configurations;
• Dedicated node pools, RuntimeClass configured with an isolation runtime, or isolated clusters: for scenarios requiring stronger isolation.

ResourceQuota operates independently of the actual cluster capacity. The sum of quotas across all namespaces can exceed the available node resources, meaning it cannot substitute for capacity planning. Additionally, namespaces share the control plane and portions of the node kernel. For tenants that do not trust each other or for workloads with strict compliance requirements, a stronger security boundary must be evaluated.

Production Readiness Checklist

Release: Rolling strategy, readiness probes, graceful shutdown, rollback and migration compatibility
Capacity: Requests/limits, peak utilization margin, HPA metrics, node scaling latency
Fault tolerance: Cross-node and cross-zone distribution, Pod Disruption Budgets, dependency timeouts, recovery drills
Security: Workload identities, RBAC, PSA, NetworkPolicy, image signing
Storage: PVC lifecycle, snapshot consistency, RPO/RTO, offsite recovery
Operations: Event signaling, change auditing, controller health, certificate and key rotation

The value of this checklist isn't in checking off items once; it's in transforming those conditions into admission policies, CI checks, continuous monitoring, and regular drills.

References

• Kubernetes, Resource Management for Pods and Containers

• Kubernetes, Horizontal Pod Autoscaling

• Kubernetes, Disruptions

• Kubernetes, Resource Quotas

• Kubernetes, Pod Topology Spread Constraints

• Kubernetes, Configuring a PodDisruptionBudget