I Ran Local LLMs on a Laptop With No GPU: Here Are the Real Numbers

Local language models (LLMs) can run on a laptop without a dedicated GPU, according to a recent report. The author tested three models on their Intel Core i7-13620H laptop with 16 GB RAM and Intel UHD graphics: qwen2.5:7b, llama3.2:3b, and nomic-embed-text for embeddings. Two applications were developed during Hacktoberfest - an exam study buddy using Retrieval-Augmented Generation (RAG) with qwen2.5:7b generating answers, and a WhatsApp order ledger that converted messy chat logs into structured data.

The study buddy app allowed users to ask questions and receive answers from their own notes using local embeddings and Chroma/SQLite. The first answer after server start took 55 seconds, while normal answers took 42 seconds and quiz questions took 35 seconds. The author reported that the 3B model was about two times faster than the 7B model, and on fresh chats it was close to the 7B in terms of field accuracy.

However, both models struggled with same-day (aaj) and Kannada (nale) chats, regardless of model size. Prompt wording improved some results, but stream tokens, loading models, shrinking prompts, and capping output were suggested to enhance performance on CPUs.

The order ledger app turned messy order chats into structured data by processing them in the background and displaying results as they finished. The author concluded that running LLMs locally is feasible, especially for tasks that are asynchronous or patient (such as studying or batch extraction), but could be challenging for chat-like interactions. Model size appeared to have less impact on narrow extraction tasks, while prompt refinement and held-out accuracy sets proved more important.