Presentation: Opening the Black Box: Critical Explainability for AI Agent Tool Selection with Kiji Inspector
Hannes Hapke shares how Dataiku’s open-source Kiji Inspector applies Sparse Autoencoders (SAEs) directly to decision tokens to expose why agents pick specific tools. Moving past superficial chain-of-thought narratives, he discusses causal ablation, layer activation extraction via patched vLLM engines, and production monitoring workflows for software architects and senior engineering leaders. By…
Hannes Hapke, a machine learning engineer and director of engineering at Dataiku, introduces his presentation on using Dataiku's open-source Kiji Inspector to understand how AI agents make decisions. Kiji Inspector applies sparse autoencoders (SAEs) to decision tokens, allowing engineers to see why agents select specific tools. Hapke explains that traditional AI explanation methods, such as prompt engineering and chain-of-thought narratives, do not provide causal insights, merely offering plausible explanations for the model's actions.
To address this, the Kiji Inspector uses sparse autoencoders to force the model to select a small set of informative features during training, enabling engineers to better understand the model's decision-making process.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
