Why real-time AI at scale is so hard

Real-time AI at scale is a challenging endeavor, with numerous obstacles that often go unnoticed until they become critical. In a recent discussion, my colleague Tim Koopmans shared some hard-learned lessons about what typically goes wrong with real-time AI at scale.

One significant issue is that latency often looks fine in testing but can skyrocket under real concurrent load. For instance, an ML-based financial trading app experienced a P99 latency spike to 3 seconds when approaching 740,000 operations per second. This issue was not due to the model itself but rather the feature lookups queuing up behind writes under load. Tail latency is a property of the architecture, not a bug that can be fixed through retries, bigger caches, or connection pool tuning.

Another problem arises when stale features cause a drop in accuracy. In Tim's case, user profile staleness exceeded a five-minute SLA target, and vector embeddings became stale. Offline evaluation metrics appeared fine, but mixing them with online data led to poor performance. The model itself remained fine, but it was making decisions based on outdated data.

Vector indexes require maintenance as well. No matter how vendors advertise, "set it and forget it" isn't a viable strategy for embeddings. Every re-embedding pass degrades the index quality, and recall rates can drop dramatically. Tim experienced recall rates as low as 42%, accompanied by ballooning query latency. Tim advises treating vector indexes like any other database index, by monitoring recall accuracy, planning for partial or batch builds, and being aware that changing similarity functions or embedding models may require starting the graph over from scratch.

Resource contention is another issue, particularly when training and serving share the same hardware. Tim had one machine performing double duty, causing GPU, RAM, and CPU competition. To avoid this, it's essential to separate the write path from the read path and ensure that training is separated from serving if resources allow.

Retraining is an inevitable part of real-time AI at scale. Every model swap requires a transition period, during which the old model is still serving stale predictions while the new one warms up. A blue-green deployment, canary releases, and running old and new models in parallel can help mitigate this issue. As the scale grows, such as with 100 million ML models, the process will become even more complex.

By understanding these challenges and employing the right practices and infrastructure choices, it's possible to avoid falling into these traps and successfully implement real-time AI at scale.