Building Zero-Latency Real-Time Audio & Vision Agents in Pure Python: Sub-10ms VAD, Instant Barge-In & Full-Duplex WebSockets

In 2026, AI underwent a major shift from turn-based text interaction to real-time, full-duplex multimodal communication. This required moving away from expensive cloud APIs and addressing latency, security and compliance concerns. The article explains how to build a sub-200ms full-duplex audio and vision agent using pure Python with no third-party cloud subscriptions.

The audio pipeline involves ingesting 16kHz mono PCM audio at a sample rate of 16kHz, processing it frame by frame with a 20ms chunk size. A root mean square energy calculation determines voice activity, with a threshold of -35 dBFS triggering a barge-in interrupt to instantly stop the AI's speech output. A circular ring buffer stores the last 150ms of audio to prevent clipping of the speaker's first word.

For vision, the agent samples keyframes from the webcam at 30FPS instead of processing every frame. It computes a perceptual hash delta between successive frames to detect changes, dropping static frames that don't exceed a 2% motion threshold. Keymotion events are captured and encoded for the LLM's visual context.

The resulting architecture allows sub-10ms voice activity detection, instant interruption of the speech synthesizer, and efficient vision processing for natural, low-latency conversational AI.