Why Axolotl Users On RunPod Lose Checkpoints Silently And How To Catch It Before Your Job Starts

Axolotl is a popular framework for fine-tuning machine learning models. However, issues can arise when running Axolotl on cloud GPU pods. A reporter discovered that on RunPod A40 pods, Axolotl checkpoints would silently fail due to insufficient disk space. The reporter set up a RunPod A40 with a 10GB network volume, installed ComputeFence and Axolotl 0.20.0, and ran computefence doctor to check system readiness.

The output showed four warnings and two issues: no Accelerate configuration and a warning about low disk space on the /workspace volume. Despite the warnings, the GPU was functional. However, Axolotl's large dependency tree and checkpoint saving process could exceed the 20GB safe threshold, causing checkpoint saves to fail mid-run and losing trained weights.

To prevent this, the reporter recommends either attaching a larger network volume or explicitly configuring checkpoints to save in the persistent volume using output_dir: /workspace/outputs. The disk check, computefence doctor, can be run before starting any Axolotl job to ensure system readiness.