瀏覽器跑 LLM 的骨感限制
直接在瀏覽器跑 LLM，既兼顧隱私又不用複雜的GPU設定，太完美了吧？ 從 WebLLM 到 Transformers.js，前端社群有一群反骨仔吹起一股「邊緣 LLM」的熱潮，可是，當真正將模型落地到使用者的瀏覽器時，第一個面對的考驗就是，WebGPU 真的有比 WASM 快嗎？ 這陣子實測的結論比想像中更加戲劇化， 500M 以下的微型模型，WASM 反而快了 12%，但是對於 3B 以上的大模型，WASM 直接變成悲劇，直接讓Chrome撞的頭破血流（Chrome 沒有頭 ＝ Headless Chrome）。 測試環境 項目 規格 機器 MacBook Air M2, 8GB RAM GPU Apple M2, 8 核心 磁碟可用空間 9.7GB 瀏覽器 Chrome 136 (Playwright headless) 框架 WebLLM 0.2.84,…
In a recent test, it was found that running large language models (LLMs) directly in browsers has its limitations. While the desire for privacy and the lack of complex GPU settings make this appealing, the reality is that browser GPU performance may not always outperform WebAssembly (WASM). When testing with models of 500 million parameters or less, WASM performed 12% faster than WebGL/WebGPU.
However, for models larger than 3 billion parameters, WASM became problematic, causing Chrome to crash (as Headless Chrome lacks a physical head). The testing environment included an 8GB MacBook Air M2 with an Apple M2 chip and 9.7GB of usable disk space, a browser version of Chrome 136 (Playwright headless), and the WebLLM 0.2.84 and @xenova/transformers 2.17.2 frameworks.
Early tests revealed that 8GB MacBook Air machines had difficulty handling the 3.8 billion parameter model, Phi-3.5-mini-instruct-q4f16_1-MLC, due to storage quota constraints. The conclusion was that while small models can run effectively in browsers, larger models face significant challenges in terms of storage capacity and performance.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — it may contain errors, so check the original before relying on it.
