How we make AI coding more cost efficient without sacrificing task quality

Output quality matters in AI coding, but efficiency stems from completing tasks swiftly and accurately. Token count alone doesn't determine efficiency. Instead, the focus should be on obtaining the correct context to advance the task. Overly concise tool responses may necessitate additional calls or rework, ultimately slowing down and increasing the cost of the task. This article explores four adjustments in GitHub Copilot designed to implement this principle:

1. Preserve valuable context while minimizing repetitive output

2. Eliminate unnecessary formatting that doesn't contribute to the task

3. Condense instructions without altering essential behavior

4. Deliver completed background work without an extra retrieval step

These modifications were initially assessed using agentic coding benchmarks offline and subsequently confirmed through controlled online experiments before being released. The examples in this article originate from the GitHub Copilot CLI. Other Copilot products, including the GitHub Copilot app and Copilot code review, leverage the same underlying framework and also benefit from these improvements.

A frequent tactic to lower agent expenses involves shortening the output of each tool call. However, a utility called RTK (Rust Token Killer), which trims shell output before an agent processes it, was tested using our agentic coding benchmarks in GitHub Copilot. Although RTK condenses certain responses, when the omitted text is crucial, the model might revisit the original output or rerun the command to retrieve the needed information.

This results in more turns and carries additional context forward. Even though the tool response was shorter, the overall task used more tokens and took longer. While this suggests that tokens per tool call is an inappropriate objective, it must be evaluated across the entire task, from the user's request to the final result.

Instead, we discovered that removing unnecessary information while preserving useful context was more effective. By analyzing benchmark runs, we found that install, build, test, and lint output frequently consists of repetitive noise, whereas source-like output and arbitrary command results often contain the information an agent requires.

This insight led to the creation of a selective output compressor, informed by RTK and similar methods. The initial version was overly aggressive, causing the model to repeat work or read the entire saved output, which increased the overall cost and reduced task success. For instance, we initially compressed git diff but later removed that filter after observing that agents reopened the original output to recover missing information. These early setbacks resulted in a three-part policy:

- Preserve source-like and arbitrary output

- Reorganize search results without losing content

- Compress repetitive noise selectively

The final version was conservative because it was supported by the evaluation results, not because we aimed to build a conservative compressor. When output is compressed, the agent can still access the complete original through a direct recovery path. In offline tasks where output compression was employed, no statistically significant decline in task success was detected, and agents rarely opened the saved original.

In an online experiment, the average cost decreased slightly without any significant regression in the tracked quality metrics. Lastly, the view tool, used by agents to read file contents into context, previously prefixed every line with a number. However, current tools match surrounding code and do not utilize line numbers. The line-number prefixes remained even though they were no longer needed.

The repeated prefixes accumulated throughout a session, consuming small amounts of tokens. Therefore, we eliminated them to optimize further.