Running NVIDIA Nemotron 3.5 Bolt E4B Without a GPU: Benchmark Results

NVIDIA introduced the Nemotron 3.5 Bolt E4B model on October 23. This model is designed for agents, capable of calling tools, following instructions, and suitable for edge hardware like laptops and Jetsons. The model card's benchmarks were tested on H100s, RTX 5090s, and DGX Spark. To assess its performance without a GPU, the author loaded the official 4-bit GGUF on an 8-vCPU Xeon VM with 16 GB of RAM and no GPU. The assistant was built to listen to WAV files, select a tool, call it, and provide spoken answers.

The model is based on the Gemma 4 E4B architecture, post-trained with reinforcement learning for tool use, instruction following, and multi-turn chat. It contains around 8B total parameters, with about 4B effective. The model is licensed under OpenMDW-1.1, with the base Gemma 4 weights under Apache 2.0. The model operates on 64K tokens.

The weights include per-layer embedding tables, transformer blocks, and token embeddings, totaling around 6.18 GB on disk. This includes 3.05 GB for per-layer embeddings, 2.40 GB for transformer blocks, and 0.71 GB for token embeddings.

Hardware-wise, the author used an 8 vCPU Intel Xeon (AVX-512, VNNI), 16 GB RAM, with no GPU. The llama.cpp CPU build and llama-server were used as an OpenAI-compatible endpoint. Speech-to-text was handled by faster-whisper, and text-to-speech by Piper with the en_US-lessac-medium voice.

Performance tests were conducted with 8 CPU cores. On average, the model generated about 11 tokens per second. The server command was simple, and the model loaded within 2.5 to 6 seconds. For a sanity check, the model correctly answered 17 × 23 as 391.

The author tested tool calling with five mock tools: get_weather, calculate, create_event, set_timer, and control_light. The system prompt provided the date (Tuesday 2026-10-27, IST) and short spoken replies. Out of 20 prompts, the model correctly identified the right tool or correctly determined no tool in 60/60 cases. It accurately processed arguments 57/60 times (95%).

The median time to generate the first token was 0.87 to 1.15 seconds, and end-to-end time per prompt ranged from 2.1 to 3.6 seconds. The generated tokens averaged at 20 per prompt.

When the thinking mode was turned on, the model remained correct in tool selection (20/20) and arguments (20/20). However, it took significantly longer, with a median end-to-end time of 13.1 seconds and generated tokens averaged at 150. This indicates a 5x slowdown for one additional correct argument out of 20. For a voice assistant on CPU, setting thinking to off and handling dates in code is recommended for efficiency. The model's first token appeared in 0.36 to 0.6 seconds, while tool-call output streamed in later.