Which GPU Do You Need for AI? A Practical VRAM Guide

The increasing popularity of open-source models has simplified the creation of products based on language, image, audio and video generation. However, selecting the appropriate GPU for the workload remains a more complicated decision. This consideration applies whether you are testing a Retrieval-Augmented Generation (RAG) assistant, serving a large language model (LLM) through an application programming interface (API), generating images with ComfyUI, or building an AI-driven video workflow.

Running a GPU with insufficient memory may result in the inability to load the model or necessitate reducing the context length, batch size, resolution, or number of frames in videos. Conversely, using a GPU with significantly more memory than necessary offers functionality but at a cost that may not be justified. There is no universal GPU requirement for "an LLM" or "an image model."

Their hardware needs hinge on factors such as the model, its precision, the inference framework, the size of inputs and outputs, and the number of requests processed simultaneously.

Virtual Random Access Memory (VRAM) is high-speed memory directly connected to the GPU. During inference, it stores model weights, input data, intermediate calculations, output data, and caches. Adequate VRAM enables the GPU to process the workload without constantly moving data through slower system memory. Hence, VRAM informs you about the capacity limits of an AI workload rather than providing a complete performance score.

A seven-billion-parameter language model stored at 16-bit precision needs around 14 gigabytes (GB) of VRAM for its weights alone. An 8-bit version requires roughly half as much memory for the weights, while a 4-bit version can further reduce the requirement. These figures are not complete fit calculations because the inference server, key-value (KV) cache, runtime libraries, and temporary tensors also use memory. As the context length and number of simultaneous requests grow, language models require additional VRAM.

Image and video models have a distinct memory profile influenced by resolution, batch size, frame count, and the number of pipeline components. A basic image workflow may fit on a modest GPU, whereas a ComfyUI graph incorporating ControlNets, upscalers, Variational Autoencoders (VAEs), or additional text encoders could demand significantly more memory.

Therefore, VRAM should be regarded as a capacity constraint rather than a complete performance indicator. Two GPUs with 24GB of memory may both fit the same workload but yield different generation speeds due to disparities in architecture, memory bandwidth, and compute capabilities.

The first step in choosing a GPU is determining whether the job fits within the available VRAM. Once this is established, speed and price can be compared. Here is a practical VRAM guide with ranges serving as starting points, which are not universal compatibility guarantees:

- **8-10GB**: Suitable for focused workloads such as speech transcription, embeddings, classification, compact quantized language models, and basic image generation. They are also useful for validating a container or checking deployment functionality before upgrading to more expensive hardware. However, their limitations become apparent when the scope expands.

- **12-16GB**: A good balance for everyday LLM and Retrieval-Augmented Generation (RAG) development. These GPUs can support many quantized 7B-class language models, RAG pipelines, and moderate image-generation workflows without the expense of higher-tier GPUs. The difference between a demo and a practical application becomes apparent at this point.

For instance, consider a company knowledge assistant using a 4-bit Mistral 7B Instruct model with an 8,000-token context and one or two simultaneous requests. An RTX 4070 or RTX 5070 with 12GB of VRAM could serve as a reasonable starting point, providing enough headroom for the inference server and cache. If the application later requires longer documents or multiple concurrent users, upgrading to a 16GB GPU would offer more flexibility.

- **24GB**: Ideal for serious prototyping and more demanding inference tasks. This tier can accommodate many 7B-8B models at 16-bit precision, larger quantized models, and advanced image pipelines without immediately transitioning to professional 48GB hardware. For example, the Mistral example illustrates how the job definition can shift the recommendation.

Loading a 7B model at 16-bit precision uses around 14GB of VRAM for the weights alone. With 24GB of VRAM, developers can explore more complex use cases without immediate hardware upgrades.

- **32-48GB**: Designed for large-model inference, high concurrency, multimodal systems, and enterprise workloads. GPUs like the RTX 5090 with 40-48GB of VRAM are suited for handling larger models, longer contexts, bigger batch sizes, and demanding media workflows. Similarly, A100 series GPUs with 80-96GB of VRAM cater to large-model inference, high concurrency scenarios, and complex enterprise applications.

In summary, the selection of a GPU for AI workloads begins with understanding the VRAM requirements of the model and the specific use case. Once you have identified the suitable memory tier, you can compare the price and expected performance of available GPUs within that range. Always remember to test the exact model, container, and settings to ensure compatibility before committing to a production configuration.