Ollama keep_alive: My Model Reloaded 214 Times in One Day
My local chat app was fast every single time I tested it, and slow every single time I actually used it. That's the tell, and I ignored it for weeks. I'd type a question during development, get a first token in under a second, ship the change. Then I'd come back after lunch, ask one thing, and sit there for eleven seconds watching a cursor blink. Same box. Same model. Same prompt. It wasn't the…
A local chat application experienced varying performance, with fast responses during testing and slow responses during actual use. The issue stemmed from Ollama, a local model loader, evicting loaded models from VRAM after 5 minutes of inactivity. This caused significant delays when reloading models from disk for each request. The reporter discovered that Ollama had three options for handling model persistence, but only two of them affected the OpenAI-compatible endpoint.
By adjusting the `keep_alive` variable to a longer duration or setting it to `-1` to prevent model eviction, the reporter was able to improve the performance of their local LLM significantly.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
