Why a System-1 Decision Model Beat a Small Local LLM on My Laptop

The author aimed to run a small local SLM (System-1 decision model) and connect it to a cloud LLM (large language model) environment. However, their initial plan did not work out as expected. They started by setting up Qwen 3.5-4B, but it performed poorly. They then switched to a smaller model, Qwen 3.5-2B, but the performance improvement was marginal. The author discovered Jev and decided to try an open-source System-1 decision model called Laya, which ran comfortably on their hardware.

Running the model on the iGPU via Vulkan was not faster due to shared DDR4 RAM with the CPU, which added extra driver overhead and pushed RAM usage to 12.5 GB. By switching to CPU threads directly, generation speed remained the same, but RAM usage dropped to 9.4 GB and CPU load settled around 25%. This left the laptop cooler and more responsive.

When asking the bot to perform real tasks, like fetching Hacker News and summarizing the top post, it took over 60 seconds to respond. The bottleneck was not typing speed but prompt evaluation time. The model spent 17.87 seconds reading the prompt and tool instructions (1,337 tokens) and another 28.88 seconds reading the web text (1,548 tokens) before generating a summary.

This made the author realize that even small text-generation models were not suitable for multi-step agent tasks when they bottleneck on hardware. They decided to stick to fast System-1 decision models locally and offload heavy reasoning to cloud models.