LoRA on a 16 GB card: what fits and what does not

The maximum model size that can be fine-tuned using LoRA on a 16 GB consumer graphics card is approximately 7 billion parameters. This is not due to the model size causing errors, but rather because larger models consume more memory than the card can handle silently, leading to significant performance degradation.

In specific testing on an RTX 5070 Ti card with a 16,303 MiB VRAM limit, a 7 billion parameter model (rank 16, sequence length 512, batch size 1) reached a peak VRAM usage of 11,897.9 MiB, staying well within the card's limit. In contrast, a 14 billion parameter model completed training without any exceptions, but its peak VRAM usage rose to 22,189.8 MiB and 17,470.9 MiB, both exceeding the card's physical limit.

This resulted in a dramatic slowdown of around 50 times compared to the 7B model, with no error messages indicating a failure.

The widespread belief that "no error" means the model fits within the memory constraints is incorrect. Windows automatically allocates system memory when a CUDA allocation exceeds the GPU's limit, causing hidden penalties and slower performance. For this specific setup, the true memory ceiling for Qwen2.5-Instruct, 4-bit base, LoRA rank 16 with a sequence length of 512 tokens and batch size 1, is 7B.

Increasing any variable beyond this point pushes the model out of the card's memory capacity, regardless of whether gradient checkpointing is enabled or not. Therefore, on this particular hardware and configuration, a 14B model does not fit into a 16 GB card, even with gradient checkpointing.