I Almost Shipped a RAG Assistant That Lied About APIs That Don't Exist
I wrote this on X a few weeks ago: I just had a very bad reminder as to the fact these LLMs are statistical parrots, I let it write code I normally wouldn't trust it to write (infra code, lots of unique behaviours) and damn I wasn't talking about my own project when I wrote that. Then StacksNG proved me right, on its own corpus, in a hackathon I'm trying to win. Ask my RAG assistant to verify an…
A reporter for a news outlet wrote an article about a recent incident involving an LLM (large language model) that was used to generate code for an African fintech assistant called StacksNG. The reporter explained that they had fed the LLM prompts asking about payment providers that were not included in its training data, such as Kuda, PalmPay, Interswitch, Paga, and OPay.
The LLM was able to generate responses that appeared to be correct, despite not having any real knowledge about these providers. The reporter discovered that the LLM was able to fabricate content that looked authentic, even when it was asked to verify an Interswitch webhook signature. To fix the issue, the reporter added a rule to the system prompt that instructed the model to check whether the specific provider mentioned in the question was actually named in the context excerpts.
If not, the model should decline to answer and not substitute another provider's information. The reporter tested the fix and confirmed that it successfully prevented the LLM from fabricating responses for the five tested providers, including Interswitch. The reporter acknowledged that there was still some uncertainty in the model's responses, particularly when the retrieved context was close to the asked-about provider in terms of similarity.
However, they considered the trade-off acceptable, as the fix had improved the reliability of the assistant.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
