Your Free Lane Is Cheap Until Rework Tax Beats the Token Bill

Free capacity can seem inexpensive at first, but hidden costs can quickly add up. This article breaks down the four main expenses that appear when a free lane turns into an expensive one.

First, queue wait time from submission to when work actually begins. This is measured as the 95th percentile of time between submitting a job and starting its execution.

Second, retry inflation due to failed attempts. This counts how many times a job is retried before ultimately succeeding. More retries mean more attempts, which increases costs.

Third, repair time for changes that required human intervention or a second pass. This includes the minutes spent merging, editing, or rolling back accepted changes.

Finally, rollback time for batches that failed. This measures the minutes spent reverting failed runs after deployment.

To properly track these costs, the article suggests logging accepted changes instead of just requests. It recommends defining three terminal states for each run: accepted (merged without edits), repaired (merged after edits), and rejected (discarded or rolled back).

An SQLite database is proposed as a simple way to create an append-only ledger tracking run IDs, lane type, submission and completion times, number of attempts, tokens used, and the repair and rollback times. The provided Python script demonstrates how to connect to the database, start runs, record attempts, and close runs with their status and associated time costs.

By focusing on the accepted changes and logging the associated time costs, teams can gain a clearer picture of the true expenses associated with free capacity, rather than just seeing a zero invoice.