11.2 Kubernetes Stateful Workloads: Identity, Storage, and Recovery

Changing a database's YAML to StatefulSet does not equate to achieving a highly available database. Kubernetes can manage Pod identity, volume mounts, and orchestration ordering; replication protocols, consistency, backup integrity, and failover mechanisms remain the responsibility of the application, operator, or managed service.

PV, PVC, and StorageClass Division of Responsibilities

Pod → PersistentVolumeClaim → PersistentVolume → Storage System
                         ↑
                   StorageClass Dynamic Provisioning Strategy

• A PVC expresses the workload's requirements for volume size, access modes, and storage class.
• A PV represents the available storage resources that can be bound within the cluster.
• A StorageClass defines the provisioner, parameters, deletion policy, and binding behavior.

Access modes, as described in ReadWriteOnce, specify constraints on how a volume can be mounted and do not equate to application-level concurrent write safety, nor do they directly indicate the storage's fault domain or performance guarantees.

For topology-constrained block storage, volumeBindingMode: WaitForFirstConsumer allows the scheduler to select a node region before provisioning the volume, reducing the risk of "Pod scheduled in zone A, volume located in zone B" leading to unschedulable workloads.

StatefulSet Provides Stable Identity

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ledger
spec:
  serviceName: ledger-headless
  replicas: 3
  selector:
    matchLabels:
      app: ledger
  template:
    metadata:
      labels:
        app: ledger
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: ledger
          image: registry.example/ledger@sha256:REPLACE_ME
          ports:
            - name: peer
              containerPort: 7000
          volumeMounts:
            - name: data
              mountPath: /var/lib/ledger
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-zonal
        resources:
          requests:
            storage: 100Gi

Pods receive stable sequence numbers such as ledger-0 and ledger-1, and are reassociated with their PVCs upon rescheduling. By default, OrderedReady is created and terminated in order; only when the application does not depend on ordering should Parallel be considered.

StatefulSet requires a headless Service to provide network identity, but a stable DNS name does not guarantee that the process is healthy. Cluster members must still handle readiness checks, split-brain protection, and rejoining of lost members.

PVC Lifecycle Defaults Favor Data Preservation

When a StatefulSet is scaled down or deleted, its associated volumes are typically not automatically removed, this is a deliberate trade-off in favor of data safety. New projects should explicitly define persistentVolumeClaimRetentionPolicy and the reclaimPolicy of the StorageClass, and clearly establish who is responsible for final confirmation when a namespace is deleted.

Before scaling out, verify that the application supports adding more members. Before scaling down, ensure the application layer safely removes replicas before modifying the replicas field. Directly reducing a three-replica database to one replica will not trigger the controller to maintain quorum.

Probes Can't Just Check Port Availability

Stateful services often need to distinguish between:

• startup: Whether log recovery, WAL replay, or cluster joining has completed;
• readiness: Whether this instance is safe to begin receiving traffic of a specific type;
• liveness: Whether the process has reached a state where a restart can safely recover it.

Labeling "temporary lag" as a liveness failure will trigger restart loops, preventing the service from ever catching up. Mixing exclusive write capabilities of the primary with general process liveness in a single readiness check will unjustly interrupt read traffic.

Snapshots Are Not a Complete Backup Strategy

CSI VolumeSnapshot can quickly capture the state of a volume, but whether the state is consistent depends on the storage implementation and the application's freeze process. A production recovery plan must answer:

1. Is it crash-consistent or application-consistent?
2. How are database logs and multi-volume systems aligned to a common recovery point?
3. Is the backup replicated to a separate account, region, or fault domain?
4. Are encryption keys, schema, configurations, and operator versions preserved together?
5. How long did the last recovery drill from an empty cluster take, and to what point in time was it restored?

A backup without a proven recovery record is merely a set of unverified objects. The RPO defines how much data can be tolerated, and the RTO defines how long a service must remain available before recovery is required, these two metrics jointly determine backup frequency, replication strategy, and recovery drill cadence.

The Value and Boundaries of Operators

A mature database operator can encode knowledge about member changes, rolling upgrades, backups, and failover, yet it also introduces risks related to CRDs, webhooks, controllers, and version compatibility. Before adopting an operator, teams should evaluate:

• The operator’s alignment with the database version upgrade matrix;
• Whether the data plane continues to function when the controller becomes unavailable;
• How to handle CRD deletion and finalizer hang scenarios;
• The failover fencing mechanism during network partitions;
• Whether backups can be restored independently of the original cluster.

If a team cannot reliably manage these operational responsibilities, a managed database service is often more dependable than simply "putting a database into Kubernetes."

On-Boarding Checklist

• Is each replica's identity and volume binding predictable?
• Do nodes, availability zones, and storage share the same failure domain?
• Will multiple replicas be evicted simultaneously during resource shortages?
• Have we exercised upgrades, scaling down, node draining, and regional outages?
• Are permissions for deleting PVCs, snapshots, backups, and keys separated?
• Have RPO/RTO values been validated through actual data recovery?

The next lesson combines resource allocation, scheduling, scaling, and interruption budgets to manage capacity and failure radius within shared clusters.

References

• Kubernetes, StatefulSets
• Kubernetes, Persistent Volumes
• Kubernetes, Volume Snapshots