Presentation: Beyond Prompting: Context Engineering for Production-Grade AI

Ricardo Ferreira, a Principal Developer Advocate at Redis, presented "Beyond Prompting: Context Engineering for Production-Grade AI" at QCon AI. He discussed his journey in building a production-grade AI application using Redis and LLMs.

Ferreira began by sharing his background in software engineering and developer education before moving into solution architecture. He introduced the Agent Memory Server (AMS) project, an open-source memory layer on top of Redis, designed to support both short-term and long-term memory for building natural human-like conversations.

The AMS project was initially developed to stress-test the open-source memory layer. Ferreira explained how he used an Alexa skill called My Jarvis, backed by a Lambda function and the LangChain4j framework, to test the AMS. My Jarvis was designed to answer complex questions using an LLM like OpenAI.

Ferreira highlighted the challenges faced when developing Alexa skills, such as the 8-second timeout for responses. Despite this, he found that the LLM could respond to a wide range of questions, but the human-like interaction was the key to a satisfying user experience. He discussed issues like context poisoning and context distraction, which occurred when too much information was provided to the LLM, leading to poor responses.

The presentation covered practical strategies for integrating long-term and short-term memory in AI applications, managing LLM token limits, mitigating context rot with reranking and semantic caching, and controlling API costs under strict latency constraints. Ferreira emphasized the importance of storytelling in understanding the practical applications and challenges of building production-grade AI systems.