A used RTX 3090 is still the best GPU for local AI, and it beats most new cards on value
This old GPU is still worth considering for local AI.
Running local large language models (LLMs) demands significant performance to match the experience of paid subscription services. While it's true that one could attempt to run an LLM on a single-board computer, the interaction would be limited and far less seamless. The Nvidia GeForce RTX 3090, however, stands out as an exceptional choice for this task, particularly with regard to value.
The RTX 3090 boasts a substantial 24 GB of VRAM, which is a vital factor for local AI. Local AI systems are constrained by both computational power and memory capacity. Both Ollama and llama.cpp, popular tools for running LLMs locally, will attempt to load as much data as possible into the available VRAM. This limitation becomes apparent when dealing with medium-sized models, as even a high-end gaming GPU with 16 GB of VRAM may struggle.
A 32B model necessitates approximately 20 GB of VRAM for a capable AI experience, which the RTX 3090 comfortably provides. The GPU's ability to accommodate the entire model in memory is crucial, but it's not the only factor influencing performance. Inferencing, which relies on context to consume the remaining VRAM, can also be a bottleneck, especially with longer conversations and more tokens.
The fact that the RTX 3090 is an older model often found on classified sites is another advantage. While it once cost around $1,000, prices have surged to approximately $1,800. This may appear steep, but considering its capabilities, it represents excellent value. For instance, the RTX 5070 Ti, which only has 16 GB of VRAM, can be found for around $2,200, while the RTX 5080 with the same amount of memory costs $2,300. The RTX 5090 is scarce, and procuring one would be a significant financial commitment.
With 24 GB of VRAM, the RTX 3090 can handle a variety of AI tasks, including image generation, transcription, embeddings, and more. One could even set up a private AI assistant on a Linux home server, accessible through a familiar ChatGPT-like interface via Open WebUI. This setup would allow for document retrieval and other useful integrations, all without the need to send prompts and files to external providers.
Although newer GPUs have been released, Nvidia's focus has shifted towards AI datacenter expansion, leaving the PC gaming community somewhat neglected. As a result, the RTX 3090, which is no longer in production, is becoming increasingly scarce. While alternative options like Intel's Arc Pro B60 or AMD's RX 7900 XTX offer 24 GB of VRAM and warranties, they lack the CUDA cores and software support that Nvidia provides.
In this context, the RTX 3090 proves to be an ideal choice for those seeking the best value for local AI applications.
Written by urgent.news from XDA Developers's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
