Grep beats LSP? Why coding agents ignore your fancier tools

A recent study comparing code-finding and editing tools revealed interesting insights about the preferences of coding agents. The research measured token use and task success when using either grep or LSP-backed semantic navigation. Initially, the agent favored grep but sometimes chose the semantic path, even when it resulted in lower success rates.

The tool's interface and output shape were crucial factors in a model's decision-making process. A model's ability to use tools effectively is not solely based on precision; it also depends on how much context the tool provides and how well the interface allows the model to use that context directly. The study also found that the codebase's noise level influenced the value of semantic navigation.

On a clean repository, semantic navigation provided no significant advantage and used more tokens than grep. However, on a noisy repository, semantic navigation improved the F1 score by 0.246 and reduced token usage by 12%. The output shape of the LSP tool mattered significantly. When the semantic navigation tool returned source text alongside location data, the agent's performance improved.

This highlights the importance of designing interfaces that are familiar and easy for models to understand, as well as providing sufficient context for the next step in the task.