A fundação esquecida do RAG: Por que você deve estudar Recuperação de Informação (Information Retrieval)

In the latest articles of this series, a recurring theme has emerged: attempting to resolve RAG (Retrieval-Augmented Generation) issues by merely swapping embedding models or adding more infrastructure is a losing battle. The mathematical foundations of the latent space are breaking, architectures are becoming overly complex, and monthly expenses are becoming unsustainable.

The primary problem modern engineers face when dealing with AI is the belief that RAG was invented in 2022. The inconvenient truth is that RAG is essentially a classical search engine with an LLM draped over the top. To stop suffering with vector databases and astronomical costs, you need to stop documenting current tools and start reading the classic: Introduction to Information Retrieval (Manning, Raghavan, and Schütze).

Here's how old and reliable Information Retrieval (IR) addresses the biggest structural problems of modern RAG.

1. Hybrid Search and BM25: Immunity Against Model Swaps

Recalling the disaster of swapping GPT-1 for DeepSeek and losing geometric compatibility of your vectors? Information Retrieval offers a vaccine for this: Lexical Search (Keyword Search). Before dense embeddings existed, the industry relied on probabilistic algorithms like BM25 (based on the classic TF-IDF). BM25 doesn't map abstract concepts in space; it maps the exact frequency and rarity of words in documents.

The magic of BM25 is that it doesn't use neural networks, so it doesn't age. It runs on disk using inverted indices (extremely cheap and consumes almost no RAM). It finds acronyms, product names, error codes, and database IDs much better than semantic search. The industry resolves vector fragility using Hybrid Search. You search in the vector (to grab semantic context) and in BM25 (to grab exact words).

If your embedding model becomes obsolete or is in the midst of re-indexing, BM25 keeps system precision deterministic.

2. Multi-Stage Architecture: Cutting Costs at the Root

In the FinOps article, we saw that comparing vector distances in 10 Terabytes of data costs a fortune in RAM. How do Google or Amazon search in Petabytes without going bankrupt? Information Retrieval teaches that information retrieval should never be a single step, but rather a funnel (Multi-Stage Retrieval Pipeline): Retriever (Fast and Cheap Retrieval): You use a BM25 in a relational database or small quantized vectors (on disk) to bring the top 1,000 most relevant documents.

It cost almost nothing. Re-ranker (Fine-grained and Expensive Ranking): You take these 1,000 documents and pass them through a heavy model (like a Cross-Encoder), which will deeply analyze the relationship between the user's query and each document, ranking the top 10 absolute. Instead of making the database calculate heavy geometry for 1 billion records, you filter with classic algorithms and only let expensive math do the last mile.

3. Evaluation Metrics: Stopping to Test in the Eye

How do you know that Phrase A matched well with Phrase B in your vector database? Most developers today ask a question in the prompt and think the answer turned out well. That's not engineering, that's guessing. Information Retrieval literature gives you the mathematical tools to prove that your search works. When you study IR, you stop measuring success by the mood of the LLM and start using: Precision at K: Out of the 5 documents the database returned, how many truly answer the question?

Recall: Out of all the good documents in your base about this subject, how many did the system manage to find? NDCG (Normalized Discounted Cumulative Gain): Was the most important document found at position 1 or position 5? Without these metrics, you'll never know if swapping GPT for DeepSeek really improved your system or if you only changed the order of errors.

In conclusion: hype passes, fundamentals remain. Learning to integrate OpenAI APIs or run LangChain takes one weekend. Building a robust inference engine that scales to Terabytes, supports architecture swaps, and doesn't financially break your company requires computer science fundamentals. Introduction to Information Retrieval was published in 2008.

Nearly two decades later, its concepts are exactly what separates a RAG prototype from a mission-critical business system. Don't invest your time learning the framework of the week. Study how search engines really work underneath the hood. The mathematics of the 1990s and 2000s is, ironically, what will save your AI project tomorrow.