{
  "id": 2823201,
  "title": "Why real-time AI at scale is so hard",
  "url": "https://urgent.news/2026/08/23/why-real-time-ai-at-scale-is-so-hard",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-23T16:00:00.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/real-time-ai-scale/"
  },
  "original_language": "en",
  "account": "Real-time AI at scale is a challenging endeavor, with numerous obstacles that often go unnoticed until they become critical. In a recent discussion, my colleague Tim Koopmans shared some hard-learned lessons about what typically goes wrong with real-time AI at scale.\n\nOne significant issue is that latency often looks fine in testing but can skyrocket under real concurrent load. For instance, an ML-based financial trading app experienced a P99 latency spike to 3 seconds when approaching 740,000 operations per second. This issue was not due to the model itself but rather the feature lookups queuing up behind writes under load. Tail latency is a property of the architecture, not a bug that can be fixed through retries, bigger caches, or connection pool tuning.\n\nAnother problem arises when stale features cause a drop in accuracy. In Tim's case, user profile staleness exceeded a five-minute SLA target, and vector embeddings became stale. Offline evaluation metrics appeared fine, but mixing them with online data led to poor performance. The model itself remained fine, but it was making decisions based on outdated data.\n\nVector indexes require maintenance as well. No matter how vendors advertise, \"set it and forget it\" isn't a viable strategy for embeddings. Every re-embedding pass degrades the index quality, and recall rates can drop dramatically. Tim experienced recall rates as low as 42%, accompanied by ballooning query latency. Tim advises treating vector indexes like any other database index, by monitoring recall accuracy, planning for partial or batch builds, and being aware that changing similarity functions or embedding models may require starting the graph over from scratch.\n\nResource contention is another issue, particularly when training and serving share the same hardware. Tim had one machine performing double duty, causing GPU, RAM, and CPU competition. To avoid this, it's essential to separate the write path from the read path and ensure that training is separated from serving if resources allow.\n\nRetraining is an inevitable part of real-time AI at scale. Every model swap requires a transition period, during which the old model is still serving stale predictions while the new one warms up. A blue-green deployment, canary releases, and running old and new models in parallel can help mitigate this issue. As the scale grows, such as with 100 million ML models, the process will become even more complex.\n\nBy understanding these challenges and employing the right practices and infrastructure choices, it's possible to avoid falling into these traps and successfully implement real-time AI at scale.",
  "summary": "Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. The post Why real-time AI at scale is so hard appeared first on The New Stack .",
  "key_points": [
    "Latency often appears fine in testing but spikes under real concurrent load.",
    "Stale features and vector embeddings cause drops in accuracy during online operations.",
    "Vector indexes require maintenance and monitoring to maintain recall accuracy."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}