What Actually Breaks When You Put a Language Model in a Customer-Facing Flow
Search for what goes wrong with a language model in production and you get a remarkably consistent answer: hallucination, drift, latency, cost, observability gaps. The list is correct. It is also written almost entirely by companies selling observability platforms, which is why every version of it ends at the same place — a dashboard. The dashboard is not the problem. The problem is that the list…
Language models in customer-facing flows break for a variety of reasons beyond hallucination, drift, latency, and cost. The most significant issue is that these models struggle to accurately reflect policy and respond appropriately to specific customer inquiries. In production, customers ask the model narrow, high-stakes questions about terms and conditions that are not reliably captured in the model's training data.
Rather than providing correct answers, the model often produces plausible-sounding responses based on generic policy documents or support knowledge, which can be treated as statements made by the company itself. This can lead to legal issues, as seen in the Moffatt v. Air Canada case, where a customer was misled by an airline's chatbot about bereavement fares.
To mitigate this, it is crucial to treat the model's responses as part of the company's official communication and ensure that only authorized personnel can provide definitive answers. Additionally, the quality of the model's responses deteriorates after launch due to changes in user input patterns and product offerings that are not reflected in the model's prompt.
To address this, the prompt should be treated as version-controlled configuration, with any changes to pricing, policy, or plans requiring updates to the assistant's context and documentation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
