The Browsing LLM's Frustrating Limitations

Running LLMs directly in the browser is perfect, as it prioritizes privacy and eliminates the need for complex GPU settings. From WebLLM to Transformers.js, the front-end community has sparked a trend of "edge LLMs." However, when deploying models in users' browsers, the first challenge is whether WebGPU is truly faster than WASM.

Recent tests have yielded surprisingly dramatic results. For miniature models under 500M, WASM is actually 12% faster. However, for large models over 3B, WASM becomes a disaster, causing Chrome to crash.

Testing Environment:

- Machine: MacBook Air M2, 8GB RAM

- GPU: Apple M2, 8 cores

- Disk available space: 9.7GB

- Browser: Chrome 136 (Playwright headless)

- Framework: WebLLM 0.2.84, @xenova/transformers 2.17.2

Initially, the test encountered a problem with the MacBook Air M2 having only 9.7GB of disk space left, which triggered the browser's Cache API physical quota limit (about 10% of available space, approximately 970MB). This limit determines the size of the model that can be loaded on the front end.

Phi-3.5 Mini @ MBA

The plan was to test the popular lightweight model Phi-3.5-mini-instruct (3.8B parameters). However, on an 8GB MacBook Air, the model couldn't even be loaded.

Phi-3.5-mini-instruct-q4f16_1-MLC requires about 2,520 MB of VRAM. When the weight file was downloaded to the browser, a QuotaExceededError occurred. Even with various caching strategies, it still didn't work.

Phi-3.5 Mini @ MStudio

To confirm if it was a storage quota issue, the test was run on an M2 Max with the same WebLLM code and Phi-3.5-mini-instruct-q4f16_1-MLC model.

Results:

- tok/s: 57.4

- Accuracy: 90.0% (9 out of 10 questions)

- Total tokens: 956

- Total time: 16.66s

The model loaded successfully, and the output was smooth, with 57.4 tok/s.

The first hurdle for running front-end LLMs on lightweight clients is often the browser's Storage Allocator, not the GPU itself.

What model can the MacBook Air run?

1. TinyLlama-1.1B

When running TinyLlama-1.1B-Chat with @xenova/transformers (v2.17.2), although the weights could be downloaded successfully, an error occurred during inference: RangeError: offset is out of bounds at Uint8Array.set.

2. SmolLM2-360M

Testing SmolLM2-360M-Instruct (q4f16) yielded:

- Speed: 38.4 tok/s

The model ran stably but had limited reasoning capabilities.

WebGPU vs WASM Performance Comparison

A fair comparison of WebGPU and WASM performance was conducted using various models on a Mac Studio.

GPT-2 (124M) was tested twice.

Results:

- Backend tok/s accuracy total time

- WASM 23.2 30% 16.39s

- WebGPU 20.4 30% 18.68s

Surprisingly, WASM outperformed WebGPU by about 12% for the 124M model.

Why did WASM win for miniature models?

- Data transfer overhead: Tokens need to be copied frequently between CPU and GPU memory, and the transfer time directly affects performance when the computation is not large enough.

- Kernel startup cost: Each GPU kernel launch has a fixed overhead that cannot be evenly distributed by small-scale parallel computing.

- WASM evolution: The mature instruction set and JIT optimization allow the CPU to efficiently handle matrix operations of the 124M scale.

3.8B Model Comparison

Using the Phi-3.5-mini-instruct-q4f16_1 model on an M2 Max, WebGPU and WASM were compared.

Results:

- Backend tok/s accuracy total tokens total time

- WebGPU 59.7 70% (7 out of 10) 490 8.20s

- WASM 0.5 60% (6 out of 10) 413 827s (13.8 minutes)

WASM was 119 times slower than WebGPU.

Why such a large difference?

A reasonable guess is that WASM runs on the CPU, and the CPU's memory bandwidth and computing power are fixed.

WebGPU, on the other hand, offloads computations to the GPU's parallel cores, and its advantage becomes more apparent as the model size increases.

For larger models like GPT-2 (124M), the kernel startup and data transfer overhead of GPU outweigh the benefits of parallelization.

A larger model, Qwen2.5-7B-Instruct (7B, q4f16), was tested on WebGPU.

Results:

- tok/s: 35.5

- Accuracy: 80% (8 out of 10)

- Total tokens: 643

- Total time: 18.13s

The 7B model ran at 35.5 tok/s on WebGPU.

To improve the Phi-3.5 Mini model on WebGPU, max_tokens was increased from 50 to 400, and accuracy improved from 70% to 100%, with tok/s stable between 59.7-65.2.

Conclusion:

- Small models do not require WebGPU: For tasks that only need miniature models (<500M), using WASM directly can avoid compatibility issues with WebGPU on different browsers and eliminate the need for complex fallback logic.

- Storage space is a hidden killer: When deploying models over 1B, prioritize checking the storage quota, as insufficient user disk space leading to cache failure often causes user experience collapse.