{
  "id": 6915625,
  "title": "6 LLM Integration Mistakes That Look Fine in a Demo and Break in Production",
  "url": "https://urgent.news/2026/09/12/6-llm-integration-mistakes-that-look-fine-in-a-demo-and-break-in",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-12T11:21:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/zunairah_bfe3d030a9be261c/6-llm-integration-mistakes-that-look-fine-in-a-demo-and-break-in-production-429b"
  },
  "original_language": "en",
  "account": "Integrating Large Language Models (LLMs) into production systems can be fraught with hidden pitfalls that only surface when real traffic begins flowing. Here are six common mistakes, along with their remedies, culled from the AI & LLM Integration Cookbook:\n\n1. Avoid conflating streaming and non-streaming functionality in a single function. Splitting calls into separate functions, each with a clear signature, avoids confusion and ensures proper handling of responses. For instance, provide one function that returns a complete response (`ask(prompt)`) and another that yields text chunks as they arrive (`ask_streaming(prompt)`).\n\n2. Ensure proper cleanup of streaming connections. Failing to close connections in the face of errors can lead to resource leaks and unexpected behavior. Use a context manager to guarantee that connections are closed even if an exception occurs mid-stream.\n\n3. Cap conversation memory correctly by slicing chats into pairs rather than by message count. This prevents orphaned messages that the API may reject, maintaining the integrity of the conversation flow.\n\n4. Normalize vectors before performing cosine similarity searches. Failing to normalize vectors can lead to inaccurate search results, as the approximation for inner product similarity only holds true when vectors are unit-normalized.\n\n5. Only retry errors that are transient. Blanket retry logic for all exceptions can exacerbate issues, especially when requests fail due to malformed parameters. Scope retries to specific exceptions, such as rate limits or timeouts, and use backoff strategies to prevent overwhelming the API.\n\n6. Provide a fallback model in addition to a retry mechanism. Relying solely on retries is insufficient for handling model degradation or downtime. Implementing a secondary, weaker model as a fallback ensures that the application can still function, albeit at a reduced capability, rather than failing outright.",
  "summary": "Most LLM code you find online works great in a Jupyter notebook and falls apart the moment real traffic hits it. The bugs aren't exotic — they're small, structural decisions that don't show up until something goes wrong at the worst possible time. Here are six of them, and the fix for each. Don't hide streaming behind a boolean flag It's tempting to write one function with an if stream: branch.…",
  "key_points": [
    "Split streaming and non-streaming functions into separate functions with clear signatures",
    "Ensure proper cleanup of streaming connections using context managers",
    "Cap conversation memory by slicing chats into pairs rather than by message count"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}