GKE Pod Snapshots Cut Model Load Times, and Move the Work to Snapshot Lifecycle Management

Google has released benchmark results demonstrating that GKE Pod snapshots can significantly reduce model load times. Startup latency can be cut by up to 89%, with a 70 billion parameter model loading in just 37 seconds and an 8 billion parameter model in 15 seconds. The feature saves the running state of a workload, including CPU and GPU memory, and restores it on demand. GKE Pod snapshots reached general availability in May for clusters running version 1.35.3-gke.1234000 or later.

Unlike caching, the snapshot holds everything the application had running, including open file descriptors, threads, CPU registers, memory, the container root filesystem, EmptyDir volumes, and tmpfs mounts. When an application is restored from a snapshot, it skips the initialization required to load the model, where most of the startup time is usually spent on large models.

However, the restore path presents its own challenges. Pod snapshots must run in GKE Sandbox, which requires the use of gVisor. Pods must run in GKE Sandbox and be using a node pool with gVisor enabled, or be in Autopilot clusters where gVisor is already available. An agent on each node manages the snapshot lifecycle, while a controller on the control plane removes obsolete snapshots. Cloud Storage holds the snapshot data.

The configuration involves two custom resources: PodSnapshotStorageConfig, which points at the bucket, and PodSnapshotPolicy, which selects Pods by label, triggers the snapshot based on workload or manual actions, and sets retention with lastAccessTimeout and a cap on snapshots per group.

Google provides an example using Codeway's Retake platform, where a custom caching layer for compiled artifacts reduced startup time from minutes to just 8 seconds. Lead DevOps engineer Ahmet Furkan Çomak noted that this reduced the time to start H100 instances for a specific job and shut them down when the job finished.

While the headline numbers showcase impressive reduction in startup times, the practical implementation of Pod snapshots involves additional considerations. The restore process may not be instantaneous, and hardware support is limited. Compatibility issues arise due to model digest, CUDA/driver version, GPU type, runtime configuration, and other factors. Snapshots are treated as immutable artifacts, and a Pod must verify the hash of the distilled Pod spec to ensure compatibility before restoring.

The rehydration process, where the application resumes from the snapshot, must handle encryption keys, certificates, environment variables, external connections, persistent volumes, and user-added iptables or nftables rules. These components may need to be recreated or reconfigured after the restore, as they reside outside the restored process and are not automatically restored.