Free AI Tiers Fail Differently. Run a Budget Burn-Down Before You Commit.

In the world of AI coding assistants, the free tier is not simply a scaled-down version of a paid tier. Instead, it represents a distinct product with its own unique failure modes. While token price is a factor, it is not the only metric to consider. The true measure of an AI coding assistant's performance is the number of tokens consumed per passing task, which takes into account both cost and quality.

To assess the viability of a free AI coding assistant, a 45-minute budget burn-down test was conducted. This test aims to determine whether the free tier can sustain a real workload. One such AI coding assistant, MonkeyCode, offers a free tier that includes 10 million tokens and a free server option. It is important to note that this article was prepared in collaboration with MonkeyCode as part of their product outreach.

The key issue with the free token price is that it can be misleading. Ten million tokens may seem like a substantial amount, but the reality is that agentic loops can significantly increase token usage. A single code change can trigger numerous model calls, and a single task may consume tens of thousands of tokens. Moreover, retries can double the burn rate. Failed runs are not free, as they still consume tokens.

To accurately gauge the performance of a free AI coding assistant, it is essential to calculate the tokens per passing task. This metric combines cost, quality, and reliability into a single figure, with lower values being preferable. Stability is also crucial, as it is better to have a slower but more reliable assistant than a fast one that frequently fails.

The burn-down test employs a fixed set of tasks, each with its own test assertion. These tasks are run three times to account for variance. Before each run, the environment is frozen, ensuring that the same repository, endpoint, and temperature are used. The experiment records three key metrics for each run: the number of prompt tokens, completion tokens, and whether the task passed or failed.

The provided Python code, named `burn_down.py`, serves as a harness for conducting the burn-down test. It is designed to be endpoint-agnostic, requiring only an adapter function that returns the prompt tokens, completion tokens, and the text generated by the AI coding assistant. The code defines a set of tasks, each with a unique identifier, prompt, and test assertion.

The `run_one` function executes a single task a specified number of times, measuring the execution time and token counts. The results are then appended to a list, which is finally printed.