Your 16 GB model needs 23 GB to run. The missing 7 GB is where most people get caught
Kavish buys hardware for a hospital group, where patient questions cannot leave the building. His first question: how much GPU memory to buy. The surprise: a model whose files are 16 GB needs about 23 GB to run. The file size is not the memory needed: the file is only the weights. Two more things fill the memory, and the second one is the one that surprises people. The suitcase: three things fill…
Kavish is purchasing hardware for a hospital group, where patient information must stay on-site. He asks about the GPU memory required for a particular model. To his surprise, the model's file size of 16 GB only accounts for 16 GB of the total memory needed. The other two factors that contribute to the total memory usage are the KV cache and overhead.
KV cache, also called the key-value cache, is essentially the model's memory of the conversation. It stores the notes about each new word, saving the model from having to re-read the entire conversation for each new token. The size of the KV cache grows proportionally with the length of the chat and the number of concurrent conversations.
Additionally, there is overhead, which consists of small gaps and buffers that are typically allocated at around 15% of the total memory needed. To illustrate, consider an 8-billion-parameter model with the following specifications: two bytes per number, chats of 4,096 tokens each, 8 chats open at once, 32 layers, 8 KV heads, a head size of 128, and a 24 GB card for comparison.
The weights alone take up 16 GB. For each token, the KV cache requires 131,072 bytes, which amounts to about 4.29 GB for all chats. Adding 15% overhead to the sum of the weights and KV cache brings the total to approximately 23.3 GB. Thus, a 24 GB GPU is sufficient to run the model, leaving about 0.7 GB of spare capacity. This amount of spare capacity is important, as it indicates that the model can run smoothly on the given hardware.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to