{
  "id": 1057491,
  "title": "Measuring the real concurrency ceiling of an LLM agent runner",
  "url": "https://urgent.news/2026/08/15/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-15T14:24:21.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/arti0/measuring-the-real-concurrency-ceiling-of-an-llm-agent-runner-53cb"
  },
  "original_language": "en",
  "account": "A reporter investigates the concurrency limits of an LLM agent runner and discovers surprising results. Initially, the reporter believed the bottleneck lay within the runner process itself. To test this theory, a benchmark was created to measure the performance of N concurrent chat sessions against the runner. The benchmark tracked four metrics: ttfa (Time to First Activity), first-Ollama (first request hitting the model queue), done (user receives the answer), and wall (total time until background work finishes).\n\nSurprisingly, the results revealed that the runner itself had no impact on performance. The event loop processed incoming POST requests without queuing, CPU usage peaked at 63%, and free RAM remained consistently available. Instead, the bottleneck was found to be Ollama, the local model server. Latency inside Ollama's internal request queue for a single loaded model (hermes3 8.0B Q4_0) caused the issue.\n\nThe reporter then examined the scheduler code responsible for managing coding jobs, which communicate with the Claude API instead of Ollama. The scheduler was hardcoded to run one job per repository at a time, a decision driven by the need to prevent merge conflicts. However, this policy limited job concurrency, regardless of the MAX_CONCURRENT_JOBS setting.\n\nTo address this issue, the reporter replaced the blind \"one job per repo\" check with a more intelligent path-overlap predicate. Two queued jobs can now run concurrently if they meet three conditions: disjoint named paths, no dependency edges, and historical touched files. This approach allows for safer concurrent execution while minimizing the risk of merge conflicts.\n\nThe story concludes by emphasizing the importance of measuring performance metrics before assuming a bottleneck exists. In this case, the perceived limit of the runner was actually the performance of the local model server. The reporter also highlights the need for smarter scheduling policies to optimize job concurrency without causing conflicts.",
  "summary": "I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck. Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling…",
  "key_points": [
    "Initial belief that runner process bottlenecked performance",
    "Benchmark revealed Ollama as the actual performance bottleneck",
    "Scheduler hardcoded one job per repo, limiting concurrency"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}