An 80GB H100, and only 3B of the model could keep up with conversation
📝 Originally published (in Japanese) at forge.workstyle.tech . I used to think that 80 GB of VRAM would be enough to chat with a large model. In reality, I could only effectively run a 3B-parameter model—and that was without voice. The full model is 35B parameters. However, only a 3B worth of weights are actively used for each generated token. It took me quite a while to realize that I had been…
I initially believed that an 80GB H100 GPU with 80GB of VRAM was sufficient to run a large language model and engage in fluid conversations. However, I soon discovered that only a 3B-parameter model could effectively function within these memory constraints. The full model, with 35B parameters, only utilized 3B weights for each generated token.
I had confused the concepts of memory capacity and usability for conversation. While the GPU could accommodate the model, the actual conversational quality depended on computation per token, which increased latency. Smaller models may reduce response time, but they often sacrifice the ability to accurately interpret the intent of the user's utterances, particularly in the context of continuations of a previous action.
The MoE (Mixture of Experts) mechanism allowed the 35B model to function with only 3B weights per token, similar to smaller models in computational cost. Despite this, accuracy in determining the type of request the user was making was higher with the larger model. In voice conversations, additional factors such as speech recognition and text-to-speech processing must be considered, further complicating the evaluation of the system's responsiveness and accuracy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
