Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices

Day 20: Persistent Storage in Kubernetes — Managing State, Weights and Vector Indices

Containers in Kubernetes are transient. When a Pod fails or Kubernetes migrates it, all data on the container's local disk is lost. This is acceptable for typical web servers, but not for artificial intelligence workloads. Vector indices and large model weights take time to reconstruct and are many gigabytes in size, so re-downloading them on every restart is inefficient. To solve this problem, Kubernetes provides persistent storage resources.

Kubernetes separates storage from the application using two resources: PersistentVolume (PV) and PersistentVolumeClaim (PVC). A PV represents a piece of real storage in the cluster, such as local NVMe disks, an NFS share or a SAN. It specifies the size, access modes and backing hardware. A PVC is the application's request for storage. The Pod references the claim, and Kubernetes binds the claim to a matching volume or, if necessary, creates a volume on demand using a StorageClass.

Access modes determine how Pods can use the volume. ReadWriteOnce (RWO) allows mounted read-write access from a single node, making it suitable for stateful vector databases like Milvus or Qdrant, where concurrent writers could corrupt the index. ReadOnlyMany (ROX) permits read-only access from multiple nodes simultaneously, ideal for model weights.

This allows one copy of large weights to serve many inference Pods instead of each Pod having its own copy. It is crucial to verify that your storage backend supports the requested access mode, as shared read-only mounts often come from NFS or similar network filesystems, while plain block disks are typically single-node.

StatefulSets are recommended for AI workloads like vector databases. Unlike Deployments, StatefulSets assign each Pod a stable name and start them in order. Each Pod is bound to its own persistent volume that persists across restarts. This ensures consistent replication between database Pods and prevents data loss during cluster updates.

A StatefulSet example involves a shared, read-only volume for model weights (requiring a backend supporting ReadOnlyMany) and a single Pod for the Qdrant vector database with its own volume per Pod. This setup eliminates repeated weight downloads, preserves vector indices, and protects system memory across cluster updates.