Podcast: Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization. By…
In this episode of InfoQ Podcasts, Edoardo Vacchi, a Red Hat engineer, discusses the evolving landscape of WebAssembly in backend environments, distributed AI inference using llm-d and vLLM on Kubernetes, and running Java workloads on GPUs via TornadoVM. Edoardo explains how llm-d manages cache locality, request routing, and batch inference across Kubernetes clusters to optimize GPU utilization.
He also touches on the WebAssembly ecosystem's evolution, Wasm's shift towards a modular system, and the growing importance of sandboxing in today's development landscape.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
