Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture

Voice Agents in Microsoft Foundry are a real-time speech-to-speech architecture designed to handle chat-based agents in a way that previous request/response agents couldn't manage. When a user talks, they don't pause for the agent to decide whether to invoke a function; they keep talking, interrupt, and expect a natural reply within a few hundred milliseconds.

To handle this, Microsoft Foundry's Voice Agents (currently in preview) provide a first-class agent kind alongside other agent types like prompt agents, hosted agents, workflows, and external agents.

The interesting engineering behind Voice Agents isn't just the addition of voice mode, but how they restructure agent execution to make tool calling, turn detection, and interruption handling work over a persistent WebSocket instead of a stateless HTTP call. This architecture article provides a deep, implementation-level look at the architecture, explaining what happens on the wire, why function calling requires a deferred-response pattern, how turn detection and barge-in actually work, and production considerations such as security, cost, scale, failure modes when using live microphones with LLMs.

Voice Agents solve the problem of speech-to-speech assistants by providing a managed realtime orchestrator that owns the wire protocol, turn-taking, and (optionally) transcript/audio persistence. They sit in the same project_client.agents management surface as other agent kinds, inheriting the same governance model, RBAC, and audit trail.

However, the runtime surface for Voice Agents is different, using project_client.agents for management, project_client.beta.voice_agents.realtime for the live WebSocket connection, and project_client.beta.voice_agents.conversations for accessing persisted transcripts and audio after the fact.

To understand the architecture better, it's essential to know that the connection is a session, not a call, and everything — user turns, model responses, tool dispatch, and persistence — happens within this session. This differs from text agents, where the connection is a call. The two key design decisions that stand out in this architecture are the session-based connection and the deferred-response pattern for function calling, which requires a different runtime model than text agents.