I built a "Can I run this LLM?" site where every VRAM number is measured, not guessed

A new website has been created to provide accurate measurements of VRAM requirements for large language models (LLMs), addressing the discrepancies found among various online sources. When Kimi K3 was released, five popular pages gave different estimates for the VRAM needed, ranging from 800 GB to 1.68 TB. The reason for these discrepancies is that the calculations often ignore important factors such as the embedding table, attention tensors in higher precision, the output head, and container overhead. These small omissions can add up to hundreds of gigabytes for a model with 1 trillion parameters.

To resolve this issue, the creator of the website, ModelFit, built a tool called fetch.py, which retrieves data from the Hugging Face API. This tool fetches trending GGUF repositories, including the top models by downloads and hype keywords. For each repository, the tool processes the tree structure, summing the quantities of all shards into a single file, and then extracts the relevant configuration information from config.json.

The build.py script then calculates the required VRAM and RAM, taking into account the weights, KV cache, and overhead. The result is a quant-by-quant table, accompanied by GPU recommendations, an interactive VRAM checker, and an FAQ section.

The website updates daily by re-fetching, re-testing, and rebuilding the data, and then deploying the new information within 24 hours. The limitations of the site are clearly stated, such as the use of a flat overhead rule for repositories without a readable config.json file, as well as the inclusion of all resident experts in memory for MoE models.

The default context size is set to 8K, and the KV cache grows linearly with the size of the model. The website also provides rough estimates for CPU inference speed, focusing on memory bandwidth rather than capacity.

ModelFit is built using three scripts, and the data is publicly available on the Modelfit website (modelfit-eight.vercel.app). If a model's information is missing or a number appears incorrect, users can submit feedback to help improve the site. The data for ModelFit comes from the Hugging Face public API, and the model sizes belong to their respective uploaders, who are part of the community responsible for the actual quantization work.