Your Free Tier Has a Concurrency Knee. Go Find It.

The free version of MonkeyCode s server offers a large allowance of tokens, along with a complimentary server. However, it is crucial to be aware of a key aspect of the service: the concurrency knee. The free endpoint does not fail by providing incorrect answers; instead, it delays delivering the right answer by eleven seconds, making it a more challenging issue to identify.

This story examines the free model access and free server option, as described by the company, without acting as a benchmark for comparison. Please note that this article was created in collaboration with MonkeyCode's product team.

When evaluating the free tier, it is essential to focus on the queue rather than the token quota. While the token meter is visible, it rarely determines whether a job will complete. Two requests with identical prompt sizes can experience a delay of a second or thirty seconds, depending on the number of other users utilizing the same free capacity at that moment.

Consequently, the focus should not be on the number of tokens available; instead, the question should be: at what concurrency does the response shape no longer belong to me? This threshold, known as the knee in capacity work, marks the point where latency begins to grow more rapidly, and errors become more frequent. Above this line, the response shape changes significantly, leading to different failure modes.

These changes require distinct retry logic approaches, and overlooking the knee could result in ineffective error handling.