JioHotstar Benchmarks Knowledge Bases for AI Coding Agents
JioHotstar evaluated 6 approaches to code retrieval for AI coding agents using SWE, CodeScale Bench, and an internal question-answering harness. Structural search performed well on concrete tasks, while semantic retrieval helped with ambiguous requirements. The findings highlight trade-offs in accuracy, token use, and tool design, underscoring the need to match retrieval tools to task types. By…
JioHotstar tested six methods for AI coding agents to navigate large codebases. The study found that structural code search worked well for clear engineering tasks, while semantic retrieval helped with ambiguous requirements. Adding retrieval layers did not always improve task completion or reduce token usage. JioHotstar examined filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an in-house RAG pipeline.
The team used subsets of SWE-bench and CodeScaleBench, plus a custom question-answering harness. Claude Sonnet was the solving model. The filesystem baseline stayed competitive on clear tasks. Sourcebot, with indexed text search, performed better on harder tasks and was less costly. On simpler tasks, retrieval added overhead without significant improvement.
Semantic retrieval was more helpful when agents had to interpret unclear problem descriptions. The research shows that retrieval methods should be chosen based on task characteristics rather than seen as interchangeable. The team also stressed the importance of tool design, with agents favoring familiar operations like text searching, reading files, and locating symbols.
JioHotstar suggests keeping filesystem tools as a baseline, using structural search for specific code tasks, and semantic indexing for vague or widely applicable requirements.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
