My local AI ran on the CPU for a week and I didn't notice
My first Ollama setup felt fast. One evening, one model, one question, a clean answer streaming back at a speed that made cloud AI feel unnecessary. A few days later, with a bigger model and a longer question, the cursor crawled. No error. No warning. Just slow, in a way that made me suspect the model, the network, my own question — everything except the actual cause, which turned out to be…
A local AI setup ran on the CPU for a week without the owner noticing. The issue was caused by oversizing the model and context length, which pushed the GPU's VRAM limit, resulting in partial offloading to the CPU. The setup, running Ollama as a service on Proxmox, was expected to keep all requests local. However, the real work was sizing the model correctly, considering the GPU available and the context length.
Two mistakes were made: using a model that partially offloaded to the CPU without any error and another VM already claiming the GPU through passthrough. To avoid such issues, it's essential to check GPU utilization after any change, assign GPU passthrough to only one VM, and ensure the GPU is fully removed from other VMs before rebooting. This ensures the AI service remains fast and local, keeping all requests within the house.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written; read the original for the full account.
