{
  "id": 9098891,
  "title": "The Shortlist Decides First",
  "url": "https://urgent.news/2026/09/22/the-shortlist-decides-first",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-22T06:21:29.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/steefjan_wiggers_34a415b/the-shortlist-decides-first-1d5a"
  },
  "original_language": "en",
  "account": "The evaluation framework for tool selection was devised by readers. It is divided into two stages, each with its own price tag. In stage one, retrieval quality is determined by matching user requests to tool names and descriptions. Retrieval quality can be measured deterministically, which is why stage one needs no model. In stage two, a live model is required, along with many runs and scripted tasks. However, the retrieval mechanism only sees what the description says. If the user speaks the tool's vocabulary, retrieval is essentially solved, even with terse descriptions scoring perfectly. However, when the user paraphrases, recall collapses to just five percent, and adding more words through verbosity does not improve retrieval. The assumption that deferred tool loading and on-demand retrieval is sufficient is flawed. Even adding more tools to the shortlist does not solve the vocabulary gap, as it only results in a linear increase in cost without improving recall. The solution lies in adding vocabulary to the descriptions. Appending one sentence with synonyms to each realistic description increased recall from five percent to 100 percent, without significantly increasing the cost. This approach is more cost-effective than expanding the shortlist, which would require shipping all hundred tools. The findings provide a budget rule for description tokens: spend them on disambiguation, provenance, and the user's own vocabulary. The alias sentence found to be the cheapest line in the contract, as it closes the vocabulary gap without incurring a high cost. The approach is applicable to any retriever, including embedding-based ones, as it enriches the surface for retrieval.",
  "summary": "The eval this post runs was designed by the readers. A commenter on What the Agent Pays for Discovery reframed tool selection as a two-stage system: first recall, whether the correct tool makes it into the shortlist that deferred loading or retrieval produces, then selection, whether the model chooses it from that shortlist. He added the design constraint that stuck with me: a cheap context with…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}