11.2 Kubernetes Stateful Workloads: Identity, Storage, and Recovery

Changing a database's YAML to StatefulSet does not equate to achieving a highly available database. Kubernetes can manage Pod identity, volume mounts, and orchestration ordering; replication protocols, consistency, backup integrity, and failover mechanisms remain the responsibility of the application, operator, or managed service.

PV, PVC, and StorageClass Division of Responsibilities

Pod → PersistentVolumeClaim → PersistentVolume → Storage System
                         ↑
                   StorageClass Dynamic Provisioning Strategy

• A PVC expresses the workload's requirements for volume size, access modes, and storage class.
• A PV represents the available storage resources that can be bound within the cluster.
• A StorageClass defines the provisioner, parameters, reclaim policy, and binding behavior.

ReadWriteOnce permits read-write mounting by one node; multiple Pods on that node may still access the volume. For cluster-wide single-Pod access, evaluate CSI support for ReadWriteOncePod. Neither implements application transactions or membership fencing, nor specifies storage failure domains or performance guarantees.

For topology-constrained block storage, volumeBindingMode: WaitForFirstConsumer delays binding or provisioning until Pod scheduling constraints can be considered, reducing the risk of "Pod scheduled in zone A, volume located in zone B" leading to unschedulable workloads. An already bound zonal volume does not move between zones as a result; zone failure still requires storage replication or backup recovery.

StatefulSet Provides Stable Identity

These manifests show network identity and volume binding, not a complete database deployment. Put both objects in the same namespace. Configure fast-zonal beforehand, replace the image placeholder with a real digest, and supply the application’s membership protocol and appropriate probes.

apiVersion: v1
kind: Service
metadata:
  name: ledger-headless
spec:
  clusterIP: None
  selector:
    app: ledger
  ports:
    - name: peer
      port: 7000
      targetPort: peer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ledger
spec:
  serviceName: ledger-headless
  replicas: 3
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  selector:
    matchLabels:
      app: ledger
  template:
    metadata:
      labels:
        app: ledger
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: ledger
          image: registry.example/ledger@sha256:REPLACE_ME
          ports:
            - name: peer
              containerPort: 7000
          volumeMounts:
            - name: data
              mountPath: /var/lib/ledger
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-zonal
        resources:
          requests:
            storage: 100Gi

Pods receive stable names such as ledger-0 and ledger-1; ledger-0, for example, uses PVC data-ledger-0. A replacement has a new UID but reuses the original PVC. Default OrderedReady creates Pods in order and terminates them in reverse order during scale-down. Deleting the whole StatefulSet does not guarantee that termination order.

StatefulSet requires a headless Service to provide network identity, but a stable DNS name does not guarantee that the process is healthy. Member discovery must handle readiness and bootstrap dependencies. If the first Pod cannot become Ready until all three members exist, OrderedReady can prevent the other two from ever being created. Use Parallel only when the application supports parallel bootstrap; separately evaluate the Service’s publishNotReadyAddresses if discovery needs unready peers.

An unreachable process may still be running. After force deletion of a Pod in an unknown state, its old process may continue writing. Fence the old member from storage or service access before replacement; the existence of a new Pod with the same name does not prove split-brain is impossible.

PVC Lifecycle Defaults Favor Data Preservation

The example explicitly retains PVCs during scale-down and StatefulSet deletion. Deleting a PVC is a separate operation: the PV’s reclaimPolicy: Delete typically deletes the backing storage, while Retain leaves it for manual reclamation. Deleting a namespace deletes its PVCs regardless of the StatefulSet retention setting. Check these deletion permissions and backups separately.

Before scaling out, verify that the application supports adding more members. Before scaling down, ensure the application layer safely removes replicas before modifying the replicas field. Directly reducing a three-replica database to one replica will not trigger the controller to maintain quorum.

Probes Can't Just Check Port Availability

Stateful services often need to distinguish between:

• startup: Whether log recovery, WAL replay, or cluster joining has completed;
• readiness: Whether this instance is safe to begin receiving traffic of a specific type;
• liveness: Whether the process is stuck in a state that requires a restart to recover.

Labeling "temporary lag" as a liveness failure will trigger restart loops, preventing the service from ever catching up. Mixing exclusive write capabilities of the primary with general process liveness in a single readiness check will unjustly interrupt read traffic.

Snapshots Are Not a Complete Backup Strategy

CSI VolumeSnapshot requires snapshot CRDs, a controller, and driver support, and commonly captures a single volume quickly. Application consistency depends on the storage implementation and application quiescing. Taking separate snapshots of several volumes does not automatically produce a common point in time. A production recovery plan must answer:

1. Is it crash-consistent or application-consistent?
2. How are database logs and multi-volume systems aligned to a common recovery point?
3. Is the backup replicated to a separate account, region, or fault domain?
4. Are encryption keys, schema, configurations, and operator versions preserved together?
5. How long did the last recovery drill from an empty cluster take, and to what point in time was it restored?

A backup without a proven recovery record is merely a set of unverified objects. RPO is the acceptable time by which the recovered data may lag the failure; RTO is the acceptable time to restore service. An RPO of 5 minutes and an RTO of 30 minutes require a recovery point at most 5 minutes before the failure and the agreed service restored within 30 minutes. Together, these constrain backup frequency, replication, and drill cadence.

The Value and Boundaries of Operators

A mature database operator can encode knowledge about member changes, rolling upgrades, backups, and failover, yet it also introduces risks related to CRDs, webhooks, controllers, and version compatibility. Before adopting an operator, teams should evaluate:

• The operator’s alignment with the database version upgrade matrix;
• Whether the data plane continues to function when the controller becomes unavailable;
• How to handle CRD deletion and finalizer hang scenarios;
• The failover fencing mechanism during network partitions;
• Whether backups can be restored independently of the original cluster.

If a team cannot reliably manage these operational responsibilities, a managed database service is often more dependable than simply "putting a database into Kubernetes."

Production Readiness Checks

• Is each replica's identity and volume binding predictable?
• Do nodes, availability zones, and storage share the same failure domain?
• Will multiple replicas be evicted simultaneously during resource shortages?
• Have we exercised upgrades, scaling down, node draining, and zone outages?
• Are permissions for deleting PVCs, snapshots, backups, and keys separated?
• Have RPO/RTO values been validated through actual data recovery?

The next lesson combines resource allocation, scheduling, scaling, and disruption budgets to manage capacity and failure radius within shared clusters.

References

• Kubernetes, StatefulSets

• Kubernetes, Persistent Volumes

• Kubernetes, Volume Snapshots

• Kubernetes, Force deletion and StatefulSet identity