Measuring the real concurrency ceiling of an LLM agent runner
I wanted to raise the concurrency limits on my local AI agent runner. The UI now supports multiple terminal panes running in flight, and my gut told me the runner process itself was becoming the bottleneck. Before touching a single config setting, I wrote a benchmark to test that assumption: When N sessions run at once, what actually breaks — the model server, the hardware, or the scheduling…
A reporter investigates the concurrency limits of an LLM agent runner and discovers surprising results. Initially, the reporter believed the bottleneck lay within the runner process itself. To test this theory, a benchmark was created to measure the performance of N concurrent chat sessions against the runner. The benchmark tracked four metrics: ttfa (Time to First Activity), first-Ollama (first request hitting the model queue), done (user receives the answer), and wall (total time until background work finishes).
Surprisingly, the results revealed that the runner itself had no impact on performance. The event loop processed incoming POST requests without queuing, CPU usage peaked at 63%, and free RAM remained consistently available. Instead, the bottleneck was found to be Ollama, the local model server. Latency inside Ollama's internal request queue for a single loaded model (hermes3 8.0B Q4_0) caused the issue.
The reporter then examined the scheduler code responsible for managing coding jobs, which communicate with the Claude API instead of Ollama. The scheduler was hardcoded to run one job per repository at a time, a decision driven by the need to prevent merge conflicts. However, this policy limited job concurrency, regardless of the MAX_CONCURRENT_JOBS setting.
To address this issue, the reporter replaced the blind "one job per repo" check with a more intelligent path-overlap predicate. Two queued jobs can now run concurrently if they meet three conditions: disjoint named paths, no dependency edges, and historical touched files. This approach allows for safer concurrent execution while minimizing the risk of merge conflicts.
The story concludes by emphasizing the importance of measuring performance metrics before assuming a bottleneck exists. In this case, the perceived limit of the runner was actually the performance of the local model server. The reporter also highlights the need for smarter scheduling policies to optimize job concurrency without causing conflicts.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
