{
  "id": 10163411,
  "title": "GKE Pod Snapshots Cut Model Load Times, and Move the Work to Snapshot Lifecycle Management",
  "url": "https://urgent.news/2026/09/27/gke-pod-snapshots-cut-model-load-times-and-move-the-work-to-snapshot",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-27T06:46:00.000Z",
  "source": {
    "name": "InfoQ",
    "slug": "infoq",
    "url": "https://www.infoq.com/news/2026/09/gke-pod-snapshots-benchmarks/"
  },
  "original_language": "en",
  "account": "Google has released benchmark results demonstrating that GKE Pod snapshots can significantly reduce model load times. Startup latency can be cut by up to 89%, with a 70 billion parameter model loading in just 37 seconds and an 8 billion parameter model in 15 seconds. The feature saves the running state of a workload, including CPU and GPU memory, and restores it on demand. GKE Pod snapshots reached general availability in May for clusters running version 1.35.3-gke.1234000 or later.\n\nUnlike caching, the snapshot holds everything the application had running, including open file descriptors, threads, CPU registers, memory, the container root filesystem, EmptyDir volumes, and tmpfs mounts. When an application is restored from a snapshot, it skips the initialization required to load the model, where most of the startup time is usually spent on large models.\n\nHowever, the restore path presents its own challenges. Pod snapshots must run in GKE Sandbox, which requires the use of gVisor. Pods must run in GKE Sandbox and be using a node pool with gVisor enabled, or be in Autopilot clusters where gVisor is already available. An agent on each node manages the snapshot lifecycle, while a controller on the control plane removes obsolete snapshots. Cloud Storage holds the snapshot data.\n\nThe configuration involves two custom resources: PodSnapshotStorageConfig, which points at the bucket, and PodSnapshotPolicy, which selects Pods by label, triggers the snapshot based on workload or manual actions, and sets retention with lastAccessTimeout and a cap on snapshots per group.\n\nGoogle provides an example using Codeway's Retake platform, where a custom caching layer for compiled artifacts reduced startup time from minutes to just 8 seconds. Lead DevOps engineer Ahmet Furkan Çomak noted that this reduced the time to start H100 instances for a specific job and shut them down when the job finished.\n\nWhile the headline numbers showcase impressive reduction in startup times, the practical implementation of Pod snapshots involves additional considerations. The restore process may not be instantaneous, and hardware support is limited. Compatibility issues arise due to model digest, CUDA/driver version, GPU type, runtime configuration, and other factors. Snapshots are treated as immutable artifacts, and a Pod must verify the hash of the distilled Pod spec to ensure compatibility before restoring.\n\nThe rehydration process, where the application resumes from the snapshot, must handle encryption keys, certificates, environment variables, external connections, persistent volumes, and user-added iptables or nftables rules. These components may need to be recreated or reconfigured after the restore, as they reside outside the restored process and are not automatically restored.",
  "summary": "Google has published benchmarks for GKE Pod snapshots, reporting up to 89% lower startup latency and a 70B model loading in 37 seconds. The feature checkpoints CPU and GPU memory through gVisor into Cloud Storage. Practitioners have asked whether invalidation is the harder problem, since snapshots match on a spec hash, machine series, and kernel and driver versions. By Steef-Jan Wiggers",
  "key_points": [
    "GKE Pod snapshots can reduce model load times by up to 89%",
    "Startup latency for 70 billion parameter model reduced to 37 seconds",
    "Snapshots hold open file descriptors, threads, memory, and filesystems"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}