Free AI Tiers Need a Stress Test. Here's the Harness

Free AI endpoints may seem like generous gifts, but most of them lack proper measurement. While a free server can save you money, it can also cause significant delays. The author has created a small harness called MonkeyCode to distinguish between retryable and blocking work. MonkeyCode offers free model access with a ten million token allowance, sufficient for real work.

The author emphasizes that free allowances are generous but not reliable. Claims are cheap, while evidence is not. To address this issue, the author designed a reproducible experiment that can be run against any OpenAI-compatible endpoint. The experiment takes less than an hour and is based on ten tasks, three trials, and four metrics.

The key to this test is your own task set, which should mirror your real workload. If you generate SQL, write SQL tasks. If you refactor TypeScript, write TypeScript tasks. Generic trivia does not provide meaningful insights. Your tests will reveal everything. To begin, create a tasks directory with one prompt file per task, keeping each prompt under five hundred tokens.

Each task has a matching test file that serves as the judge. The model never sees the test file. The author provides an example task and its corresponding test file in the provided source. The runner script posts the task to the OpenAI-compatible endpoint, running each task three times. The response is stored for later inspection, serving as evidence of failures rather than rumors.

Finally, the checker script extracts the code from the response, runs the real test suite, and prints one word per trial to indicate pass or fail.