Five Ollama Settings You Should Tune Before Running Local Models Seriously

When running local models with Ollama, several settings can be tuned to improve performance and efficiency. The first setting is Flash Attention, which speeds up attention computation by keeping data in GPU cache instead of main memory. This is already enabled by default on systems that support it, but it can be forced on using an environment variable for guaranteed performance improvements in long-context work.

The second setting to consider is the KV cache type. This controls how the model stores key/value state during computation. However, compression only works when Flash Attention is already enabled. By compressing the KV cache to a quantized format, the model can hold a roughly double context length before running into memory limitations, at the cost of a small accuracy decrease. This trade-off may be worth it for precision work, but it should be benchmarked before committing to this setting.

The third factor to tune is the context length. While the default length is often set low to conserve memory, users working with long documents or large code files may need to increase it. However, a larger context length comes with increased memory usage, so starting with a size that matches the specific workload is recommended. Overestimating the required context can lead to slower responses as the system struggles to keep up with the additional memory demands.

The fourth setting involves deciding which layers of the model should be allocated to the GPU. By default, the system automatically decides which layers run on the GPU, but limited VRAM may necessitate explicitly setting the layer count. This split inference approach allows more of the model to run on the GPU, albeit at the expense of a slower response time. This option is particularly useful when working with larger models and limited VRAM.

Lastly, preloading the model into memory can be beneficial when making the first call to Ollama. By loading the model after an empty prompt, subsequent requests will be as fast as the rest of the operations. This technique is suitable for machines with ample memory spare, but excessive memory usage should be avoided when running multiple models simultaneously.

To implement these settings effectively, it is crucial to measure their impact on your specific workload. Flash Attention's behavior may vary across different GPUs, so benchmarking on your own hardware before committing to the change is advisable. Additionally, remember that KV cache compression can reduce accuracy for precision tasks, so test the impact on your work before applying the setting.

Keep in mind that the optimal values for context length and GPU layer allocation will differ depending on your machine and workload, so systematic measurement is essential when fine-tuning these settings.