Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Your 16 GB model needs 23 GB to run. The missing 7 GB is where most people get caught

Kavish buys hardware for a hospital group, where patient questions cannot leave the building. His first question: how much GPU memory to buy. The surprise: a model whose files are 16 GB needs about 23 GB to run. The file size is not the memory needed: the file is only the weights. Two more things fill the memory, and the second one is the one that surprises people. The suitcase: three things fill…

Kavish is purchasing hardware for a hospital group, where patient information must stay on-site. He asks about the GPU memory required for a particular model. To his surprise, the model's file size of 16 GB only accounts for 16 GB of the total memory needed. The other two factors that contribute to the total memory usage are the KV cache and overhead.

KV cache, also called the key-value cache, is essentially the model's memory of the conversation. It stores the notes about each new word, saving the model from having to re-read the entire conversation for each new token. The size of the KV cache grows proportionally with the length of the chat and the number of concurrent conversations.

Additionally, there is overhead, which consists of small gaps and buffers that are typically allocated at around 15% of the total memory needed. To illustrate, consider an 8-billion-parameter model with the following specifications: two bytes per number, chats of 4,096 tokens each, 8 chats open at once, 32 layers, 8 KV heads, a head size of 128, and a 24 GB card for comparison.

The weights alone take up 16 GB. For each token, the KV cache requires 131,072 bytes, which amounts to about 4.29 GB for all chats. Adding 15% overhead to the sum of the weights and KV cache brings the total to approximately 23.3 GB. Thus, a 24 GB GPU is sufficient to run the model, leaving about 0.7 GB of spare capacity. This amount of spare capacity is important, as it indicates that the model can run smoothly on the given hardware.

Rewritten by urgent.news; may contain errors — check the original.

Source: dev.to

Read the original at dev.to →

Report a problem with this story

More in Tech

CSV export for a time card: the format payroll people actually open

When I built the time card calculator on mytimecardcal.com , the export button was an afterthought. The calculator already did the hard work: it added up the hours, split regular time from overtime…

  • CSV export added to mytimecardcal.com calculator
  • CSV file format with clear column headers and rules
  • Download process creates Blob object for CSV string

I Generated 450+ Business Checklist Items with Python (reportlab + One Data Structure)

I sell a bundle of 30 business checklists (~450 items). People assume a team wrote them. No — a Python script did, in an afternoon.

  • Programmer generates 450+ business checklist items using Python script
  • Checklists consist of three sections with 15 items each, totaling 30 checklists
  • Python data structure serves as single source of truth for PDFs, SEO pages, and previews

More from Sunday 22 November →