Tool Selection at Scale: What Top-K RAG Leaves Out
Embed, retrieve, let the LLM decide works for about 100 tools. At 1,000 it needs a control plane: hybrid retrieval, hard filters, a learned reranker, progressive disclosure and per-stage evals. This is an adapted edition of the original article. It preserves the section argument and evidence; use the source link for the published version and any interactive material. Read the original article 01…
The article discusses the limitations of the "embed, retrieve, decide and validate" pipeline when dealing with a large number of tools, specifically at the 1,000 tool scale. The author argues that retrieval becomes an essential part of the process rather than just an optimization step. They compare this to the baseline pipeline's limitation to a single top-10 embedding lookup, which rarely misses at 100 tools but becomes necessary at 1,000 tools. The key points include:
1. Tools are not just categories but have side effects such as sending emails, deleting records, or charging cards. Tool selection involves more than simple category classification.
2. Retrieval helps control the context and improve accuracy, rather than just serving as a knowledge source. Position effects in long contexts can negatively impact accuracy.
3. The recall chain is not a single number but the product of conditional recalls at each stage: retrieval, filtering, reranking, and selection. Improving retrieval can significantly boost overall performance.
4. The focus should be on controlling the decision space, exposing the smallest context that still contains the right tool, rather than just maximizing retrieval recall.
5. Measuring performance at each stage of the pipeline is crucial to identify which stage is the weakest link and requires tuning.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
