{
  "id": 13849619,
  "title": "The Runtime IS the Benchmark: Endurance Metrics for Local Agents",
  "url": "https://urgent.news/2026/10/12/the-runtime-is-the-benchmark-endurance-metrics-for-local-agents",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-12T02:14:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ab371/the-runtime-is-the-benchmark-endurance-metrics-for-local-agents-3kb6"
  },
  "original_language": "en",
  "account": "In a recent experiment, I explored how the runtime of locally-run models compares to cloud-based options. To do this, I utilized a Gemtek NUC M6 Ultra mini-PC running Pop!_OS with zRAM swap. My goal was to benchmark sessions that could run for extended periods, such as days. I selected three models to test: a 1.2B triage model, an 8B-class model, and a cloud workhorse, all running on the same hardware with identical configurations.\n\nAfter running numerous tests, it became clear that memory systems have a hang threshold. Specifically, I discovered that this threshold is based on the number of files rather than the total file size. Near 1,500 files per store, my context assembly began to stall. To address this, I partitioned the context into 80 bounded stores, resulting in a 2.6ms/file average and a significant improvement of 3.1 times over a monolithic context.\n\nI also approached sandbox containment differently by scoring the blast radius using memory pressure, PID-seconds survived, and host RSS delta under zRAM. This revealed that mid-context facts were less likely to be used than those at the beginning or end of the context, leading to a U-shaped recall curve. Consequently, I modified my approach to write internal documents with conclusions first and receipts last, which proved beneficial for processing dense documents.\n\nFor extended sessions lasting up to 10 days, I continuously logged latency drift, memory growth, and retrieval decay. Over 30 days, I observed that agents do not fail immediately at the start but rather gradually deteriorate over time. Approximately 10% of my agent tasks required the top-tier model, while the remaining 90% showed no measurable quality loss despite a 30-fold cost difference across tiers. By measuring cost per completed task rather than cost per token, I found that the runtime is the key benchmark, emphasizing the importance of minimal notebook setups in achieving optimal performance.",
  "summary": "What I Benchmarked I tested whether the runtime around the model holds up: retrieval endurance, memory scaling, sandbox containment, context behavior. I used a Gemtek NUC M6 Ultra mini-PC on Pop!_OS under zRAM swap. I wanted to benchmark sessions that run for days. Models Tested I chose the models I actually run locally. Same runtime, same tests, three price points: a local 1.2B triage model, a…",
  "key_points": [
    "Runtime comparison between local and cloud models revealed memory-based hang threshold.",
    "Partitioning context into 80 stores improved performance by 3.1 times over monolithic context.",
    "Runtime is the key benchmark, emphasizing minimal notebook setups for optimal performance."
  ],
  "editors_take": "Shifting the focus to runtime and task completion costs reveals that smaller local models can match high-tier performance for most tasks, making minimal hardware setups viable for long-term agent use.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}