A Runaway Prompt Is a Tenant: Fair Scheduling on a Shared Free Endpoint

A runaway prompt can cause problems on a shared free endpoint, according to an article prepared as part of MonkeyCode's product outreach. Three agents, each with a separate function, shared a single endpoint, API key, and context quota. When one agent entered a retry loop after a schema validation error, it consumed the shared context budget and connection pool, causing the other two agents to fail.

The SDK does not protect against this issue, as it sees only one request at a time and has no concept of tenants, budgets, or sibling agents' needs. The only layer that can detect and isolate tenants is the orchestration layer, but most prototypes lack this layer.

To address this problem, a scheduler can be inserted between the agent loops and the endpoint. This scheduler should enforce three rules: a context budget per tenant, a fair queue that round-robins across tenants, and a stall detector that pauses a tenant after too many low-information responses. This approach limits the blast radius of a runaway prompt and allows healthy tenants to make progress. However, it trades raw throughput for bounded blast radius, and long-running tasks may require checkpointing.

The scheduler consists of about eighty lines of code and can be implemented with simple per-tenant context budgets, round-robin fair queues, and stall detection. Decisions to use this approach should consider the tradeoffs, such as the need for checkpointing for long-running tasks and the importance of tuning the budget per workload. Additionally, free quotas may change without notice, so it's essential to verify current token allowances and server terms in the project README before implementing this pattern.

To validate the scheduler, run two agents on the same endpoint, one healthy and one deliberately stuck in a retry loop. The scheduler should keep the healthy agent responsive, while removing the scheduler will result in the unhealthy agent consuming all available resources.