LLMs on Consumer Hardware — Part 2: Prefill and the Failure of the AI PC
Part 1 established the hardware, the runner, and the primary model. This entry covers what governs inference on that hardware — the two phases of inference, the cost of long context, and the cost of loading a model from disk — and compares the local machines against a free-tier cloud model. The two phases of inference Inference has two phases. Prefill processes the input prompt before any output…
Inference on consumer hardware involves two phases: prefill and generation. Prefill processes the input prompt and is compute-bound, requiring a GPU. Generation produces output tokens one at a time and is memory bandwidth-bound. The cost of prefill is significant when prompts grow large and is highly variable across different machines. Generation, however, varies less across machines.
Machine specifications include a CPU, RAM, GPU (VRAM), and storage. The primary desktop machine has a CPU of 5950X with ~80 GB DDR4 RAM, a GPU of RX 6900XT with 16 GB VRAM, and NVMe storage with a read speed of ~2.1 GB/s. The secondary box machine has a CPU of 5600G with 32 GB DDR4 RAM, a GPU of GTX 1060 with 6 GB VRAM, and SATA SSD storage with a read speed of ~0.35 GB/s. The laptop machine has a CPU of 8840U with 32 GB DDR5 RAM, no GPU, and NVMe storage with a read speed of ~2.4 GB/s.
Prefill rates vary significantly across machines, from 20 tok/s on the laptop to 360 tok/s on the primary desktop. Generation rates vary less, from 10.0 tok/s on the laptop to 18.3 tok/s on the primary desktop. Large-prompt workloads are dominated by prefill time, making machine performance crucial for such tasks. Model-load time is influenced by storage, with slower storage leading to longer cold model load times.
A notable example is a secondary box machine with a SATA SSD that takes 50 seconds to load an 18 GB model, compared to eight seconds on an NVMe drive. This longer load time can cause intermittent timeouts when models unload between calls. To mitigate this, a long keep-alive (OLLAMA_KEEP_ALIVE=24h) with pre-warming is recommended, especially on slow-disk nodes.
The "AI PC" labeled as such fails to meet expectations. The laptop with an AMD 8840U NPU rated at up to 16 TOPS fails to provide adequate prefill rates for large-model inference. CPU prefill rates are insufficient for large prompts, requiring eight to twelve minutes before a single token is produced. This contradicts the advertised TOPS and the performance of the primary desktop machine with a GPU.
Long context is paid for in memory, as the KV cache grows linearly with context length. The runner defaults to a 4K-8K window, but custom Modelfile configurations (num_ctx 65536) were tested. A 64K window was found to be stable, with a KV cache size of ~926 MiB (GPU) and 231 MiB (CPU). Larger contexts were deemed unstable due to slower prefill and potential crashes on lower VRAM-capable GPUs.
When comparing local machines to a free-tier cloud model (Gemini 3 Flash), the cloud model provides a significantly faster end-to-end latency (~5.8s) for a 150-word reasoning prompt. The local machines, however, require more time, ranging from ~20.6s to ~64.5s, depending on the hardware. While the cloud model performs more work, it demonstrates better latency performance for routine, high-volume tasks that are private, unmetered, and have predictable latency.
These findings emphasize the importance of GPU prefill for local inference, the impact of memory cost for context length, and the significance of placement based on task requirements. The next entry will explore whether multiple models can coexist without contention in memory.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
