I described 1,245 tables with an LLM and retrieval got worse

An LLM was used to generate 1,245 table descriptions, which were then indexed alongside table names to improve retrieval. However, the recall dropped significantly, with the "contacts" table falling below rank 40 in search results. The issue lies in the correlated vocabulary between table descriptions and user queries. When the descriptions are generated, they use the same vocabulary as the user queries, causing a phenomenon called IDF collapse.

This makes the most frequent terms, like "contacts," less informative and lowers their scores. Additionally, length normalization penalizes long documents, making tables with many columns, like "contacts," appear more relevant. This creates a compounding effect, where IDF collapse removes the signal that separates relevant tables from others, and length normalization further prioritizes the most important tables.

Cataloguing, while useful, adds correlated noise that attacks the very queries it aims to help. The optimal solution is to score fields separately and combine the rankings. This approach involves creating three separate BM25 indexes: one for the entire document, one for the table names, and one for the generated descriptions. By scoring the fields independently and combining the results, the model can better differentiate between relevant tables and minimize the impact of correlated noise.