The Runtime IS the Benchmark: Endurance Metrics for Local Agents

In a recent experiment, I explored how the runtime of locally-run models compares to cloud-based options. To do this, I utilized a Gemtek NUC M6 Ultra mini-PC running Pop!_OS with zRAM swap. My goal was to benchmark sessions that could run for extended periods, such as days. I selected three models to test: a 1.2B triage model, an 8B-class model, and a cloud workhorse, all running on the same hardware with identical configurations.

After running numerous tests, it became clear that memory systems have a hang threshold. Specifically, I discovered that this threshold is based on the number of files rather than the total file size. Near 1,500 files per store, my context assembly began to stall. To address this, I partitioned the context into 80 bounded stores, resulting in a 2.6ms/file average and a significant improvement of 3.1 times over a monolithic context.

I also approached sandbox containment differently by scoring the blast radius using memory pressure, PID-seconds survived, and host RSS delta under zRAM. This revealed that mid-context facts were less likely to be used than those at the beginning or end of the context, leading to a U-shaped recall curve. Consequently, I modified my approach to write internal documents with conclusions first and receipts last, which proved beneficial for processing dense documents.

For extended sessions lasting up to 10 days, I continuously logged latency drift, memory growth, and retrieval decay. Over 30 days, I observed that agents do not fail immediately at the start but rather gradually deteriorate over time. Approximately 10% of my agent tasks required the top-tier model, while the remaining 90% showed no measurable quality loss despite a 30-fold cost difference across tiers.

By measuring cost per completed task rather than cost per token, I found that the runtime is the key benchmark, emphasizing the importance of minimal notebook setups in achieving optimal performance.