{
  "id": 19633385,
  "title": "Restoring a Vertex AI Training Cluster: Recovering a Stopped GPU Node from Expired Reservations",
  "url": "https://urgent.news/2026/11/12/restoring-a-vertex-ai-training-cluster-recovering-a-stopped-gpu-node",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-12T10:04:23.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/aimldl/restoring-a-vertex-ai-training-cluster-recovering-a-stopped-gpu-node-from-expired-reservations-1pei"
  },
  "original_language": "en",
  "account": "Google Cloud's Vertex AI Training Cluster is a powerful environment for large-scale AI/ML model training. When a GPU node within this cluster experiences an expired reservation, the node enters a TERMINATED state, halting all training operations. In this scenario, the node vtctest001-a4-0, which was originally provisioned using a DWS Calendar Mode reservation, became inaccessible due to the expired reservation. Attempts to restart the VM or unbind the reservation failed due to capacity restrictions and invalid reservation IDs.\n\nTo recover the node, the DWS Flex-Start provisioning model is employed. Phase 1 involves configuring environment variables to match the existing cluster setup. The project ID, zone, and specific GPU node name are defined, ensuring that all subsequent commands target the correct resources. Phase 2 focuses on safely deleting the VM instance while preserving its boot disk, which contains the operating system, Slurm configuration, and local data. The boot disk is then snapped for safekeeping.\n\nWith the boot disk snapshot secured, the VM instance can be deleted without concern for disk retention flags. The deletion command is executed, and the user is prompted to confirm the action. Despite an error indicating insufficient permissions, the node is successfully removed from the cluster.",
  "summary": "Restoring a Vertex AI Training Cluster: Recovering a Stopped GPU Node from Expired Reservations Cluster Architecture Google Cloud’s Vertex AI Training Cluster (VTC) is designed for large-scale AI/ML model training. The environment discussed here is a High-Performance Computing (HPC) cluster provisioned using VTC. For example, the following screenshots show a cluster named vtctest001. Click the…",
  "key_points": [
    "Vertex AI Training Cluster uses GPU nodes for AI/ML training.",
    "Expired reservation caused vtctest001-a4-0 node to TERMINATED state.",
    "DWS Flex-Start model recovered node by preserving boot disk."
  ],
  "editors_take": "Leveraging the DWS Flex-Start provisioning model to recover data from terminated nodes provides a workaround for capacity restrictions and reservation expirations that otherwise halt large-scale machine learning training operations.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}