The Day My Lecture Notes Bot Contradicted Itself

On a late-night stretch of 2 AM, the reporter found themselves struggling through seventeen PDFs that resisted providing any meaningful information. With a midterm approaching in six days, the reporter's notes were a jumbled mess of arrows, acronyms, and half-typed definitions. The reporter yearned for a chatbot that could respond to questions regarding their own lectures, rather than a sophisticated model. Thus, the reporter set out to build one.

Utilizing MonkeyCode for free model access and server space, the reporter crafted a prototype that extracted text from PDFs, divided it into chunks, used a simple similarity search to retrieve the most relevant chunks, and asked a model to generate answers based on those chunks. With minimal Python code and a single POST request, the reporter's creation came to life.

The initial phase of extracting text proved relatively effortless, as most of the slides contained text-heavy content. However, one deck presented a challenge due to its rotated pages, which initially served as a warning sign of the potential issues that lay ahead. After chunking the text at a size of 1,200 characters with an overlap of 100 characters, the reporter implemented a TF-IDF vectorization and cosine similarity search to retrieve the most pertinent chunks.

Initially, the chatbot performed admirably, recalling details that the reporter had long forgotten and accurately citing slide numbers. The reporter grew increasingly confident in the model's abilities. However, after three days of testing, the reporter posed a question: "Is gradient descent guaranteed to find the global minimum?"

To the reporter's dismay, the chatbot responded confidently: "Yes. According to slide 8, gradient descent always converges to the global minimum." This assertion proved to be false, as slide 8 only stated that gradient descent converges to the global minimum for convex functions. Slide 29, on the other hand, highlighted the same concept for non-convex problems, explicitly stating that only a local minimum is guaranteed.

The chatbot had erroneously disregarded slide 29, relying solely on slide 8 to construct its answer, resulting in an overconfident and misleading response.

Recognizing the inherent risks of the chatbot's reliance on a single source of information, the reporter introduced a contradiction check to address this issue. The heuristic involved retrieving the top three chunks and comparing their textual similarity. If the top two chunks exhibited significant disagreement, the reporter flagged the response with a warning, acknowledging the potential for conflicting information.

Despite the added precaution, the reporter's experiment reinforced a critical lesson about Retrieval-Augmented Generation (RAG). The retrieval step ultimately determines the information the model can provide. If the retrieval process is biased, favoring only one side of a contradiction, the model will dutifully present that information as factual truth. The reporter concluded that refining the retrieval policy was essential to mitigating such issues, rather than solely focusing on improving the prompt structure.