Presentation: From Thousands to One: Building LLM-Powered Selection Systems

Jendrik Jördening discusses practical engineering strategies for integrating large language models (LLMs) into production pipelines. He highlights the challenges of non-determinism, schema restrictions, separating semantic text extraction from deterministic code, and validating choices using discriminator models. Jördening recommends structuring LLMs using an MVC approach to maintain database integrity, observability, and system reliability.

Jördening, who is the CTO at Nooxit, shares his experience working on ML algorithms, MLOps infrastructure, and managing internal Kubernetes clusters. He formerly worked on data science and deep learning for industry 4.0 and autonomous machines at Aurubis and Akka Germany.

The InfoQ Dev Summit Munich conference features over 20 senior software developers sharing real-world technical insights on critical software development challenges. During his presentation on building LLM-powered selection systems, Jördening realized that he needed to rework the entire talk. He explored the idea of using AI to create visuals from his bullet points, but found that the technology struggled to accurately represent text in images.

Jördening's plans to automate claim processing for Deutsche Bahn train delays led him to integrate APIs for fetching timetables and checking for changes. He discovered that AI could identify the correct EVA station number but struggled with maintaining randomness and handling prompt injections, which could have compliance implications. Despite these challenges, Jördening emphasized the importance of setting the temperature to 0 when using LLMs to ensure consistent and reliable results.