Presentation: Adaptive Recommenders in the Real World: Inference, Evals, and System Design

Mallika Rao, a former Engineering Leader at Twitter, Walmart, and Netflix, is delivering a presentation on Adaptive Recommenders in the Real World: Inference, Evals, and System Design at QCon AI. In her talk, Rao emphasizes that the true complexity of adaptive recommendation systems lies outside of model architecture, but rather in the real-time feedback loops, retrieval freshness, multi-stage orchestration, and end-to-end latency budgeting required to enable systems to continuously learn and evolve in production under real-world operational constraints like latency, cost, and observability.

Rao states that many people assume that recommendation systems are primarily focused on models, ranking, retrieval systems, embeddings, and offline metrics. However, she argues that the most challenging aspect of building recommendation systems is not just about building the model, but rather about building the entire system and every aspect of end-to-end orchestration that can continuously learn, adapt, and evolve while operating under real-world constraints.

She notes that feedback loops, evaluation, online learning, agentic behavior, and operating under uncertainty have been constant topics for recommendation systems for years, even before generative AI became mainstream. Rao believes that the lessons learned from recommendation systems are now applicable to broader AI systems, which is why she is focusing on adaptive systems in her presentation.

According to Rao, recommendation systems have evolved from handcrafted heuristics to AI-native architectures that leverage embeddings, contextual signals, multistage retrievals, and ranking pipelines to understand semantic challenges, adapt to changing user intent, and scale. However, these advancements have also introduced new operational challenges, particularly in the face of quickly evolving user behavior.

She introduces the mental model of a recommendation request, where a user arrives, candidates are retrieved, ranked, and then inferred upon, with outcomes evaluated and returned to the system as the feedback loop. Rao stresses that recommendation systems should not be treated as isolated models, but rather as feedback systems and distributed systems with increasing experimentation and adaptiveness at the boundaries between components.

Rao concludes by stating that the complexity lies in the coordination between components, rather than within each component individually. She will be discussing the adaptive aspects of recommendation systems, as well as the architecture, ranking, retrievals, and real-time inference in production. Finally, Rao will provide case studies to illustrate these concepts.