Kubernetes 1.36 restores a lost guarantee for database backups

On a late night, a PostgreSQL cluster was being restored from a backup taken the previous evening. The data directory resided on one PersistentVolumeClaim (PVC), while the write-ahead log lived on another. Despite all volume snapshots succeeding, Postgres failed to start because the write-ahead log on one volume referenced pages that were not captured in the data files on the other.

This inconsistency occurred even though the backup appeared undamaged. The problem was a lost guarantee that once existed when using traditional storage, called a consistency group. In cloud-native storage, this was lost when moving to the Container Storage Interface (CSI), which standardized snapshots around a single object, the VolumeSnapshot, scoped to a single PVC.

For a stateless service with one volume, the per-PVC model worked fine. However, for stateful workloads with state spread across multiple volumes, such as databases with separate data and log disks, the per-PVC model could not accurately determine which volumes belonged together. This led to backups that appeared healthy individually but collectively represented an inconsistent state.

To address this, Kubernetes v1.36 introduced VolumeGroupSnapshot as a generally available feature, effectively bringing back the consistency group as a first-class, vendor-neutral Kubernetes API.

A VolumeGroupSnapshot consists of three objects: a VolumeGroupSnapshotClass, defined by an administrator, describes how group snapshots are created for a given CSI driver; a VolumeGroupSnapshot, the user's request, carrying a label selector that picks out every PVC belonging to the application; and a VolumeGroupSnapshotContent, tracking the provisioned result.

The CSI driver takes one atomic, point-in-time snapshot across every selected volume, eliminating the need for application quiescence, as long as the underlying storage supports it.

The label selector is the key design choice, allowing administrators to describe groups of PVCs rather than enumerating individual volumes. For example, a selector like `app=postgres` would pick up the data and log PVCs together, preserving the consistency group boundary as volumes are added or resized. When implementing VolumeGroupSnapshot support in Velero, the core change was transitioning from "iterating over PVCs and snapshotting each" to "grouping the PVCs that belong together, snapshotting the group as one operation, then tracking the per-volume members for restore."

Not all workloads require this consistency guarantee. Individual PVC snapshots are still suitable for single-volume workloads and volumes that are truly independent. However, for workloads where cross-volume write ordering is critical, such as data plus WAL or data plus index, VolumeGroupSnapshot should be used. This decision guide helps administrators determine when to use VolumeGroupSnapshot versus individual VolumeSnapshots.