{
  "id": 8845612,
  "title": "Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture",
  "url": "https://urgent.news/2026/09/21/voice-agents-in-microsoft-foundry-inside-the-realtime-speech-to",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-21T04:56:55.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/monuminu/voice-agents-in-microsoft-foundry-inside-the-realtime-speech-to-speech-architecture-5f4b"
  },
  "original_language": "en",
  "account": "Voice Agents in Microsoft Foundry are a real-time speech-to-speech architecture designed to handle chat-based agents in a way that previous request/response agents couldn't manage. When a user talks, they don't pause for the agent to decide whether to invoke a function; they keep talking, interrupt, and expect a natural reply within a few hundred milliseconds. To handle this, Microsoft Foundry's Voice Agents (currently in preview) provide a first-class agent kind alongside other agent types like prompt agents, hosted agents, workflows, and external agents.\n\nThe interesting engineering behind Voice Agents isn't just the addition of voice mode, but how they restructure agent execution to make tool calling, turn detection, and interruption handling work over a persistent WebSocket instead of a stateless HTTP call. This architecture article provides a deep, implementation-level look at the architecture, explaining what happens on the wire, why function calling requires a deferred-response pattern, how turn detection and barge-in actually work, and production considerations such as security, cost, scale, failure modes when using live microphones with LLMs.\n\nVoice Agents solve the problem of speech-to-speech assistants by providing a managed realtime orchestrator that owns the wire protocol, turn-taking, and (optionally) transcript/audio persistence. They sit in the same project_client.agents management surface as other agent kinds, inheriting the same governance model, RBAC, and audit trail. However, the runtime surface for Voice Agents is different, using project_client.agents for management, project_client.beta.voice_agents.realtime for the live WebSocket connection, and project_client.beta.voice_agents.conversations for accessing persisted transcripts and audio after the fact.\n\nTo understand the architecture better, it's essential to know that the connection is a session, not a call, and everything — user turns, model responses, tool dispatch, and persistence — happens within this session. This differs from text agents, where the connection is a call. The two key design decisions that stand out in this architecture are the session-based connection and the deferred-response pattern for function calling, which requires a different runtime model than text agents.",
  "summary": "Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture (and Why Function Calling Is Harder Than It Looks) Why this matters Every chat-based agent you've built so far has had the luxury of a request/response boundary. A user sends a message, your agent thinks for however long it needs, calls a tool, thinks some more, and returns an answer. Nobody is standing there in…",
  "key_points": [
    "Voice Agents provide real-time speech-to-speech architecture for chat-based agents.",
    "Architecture uses persistent WebSocket connection instead of stateless HTTP calls.",
    "Deferred-response pattern and turn detection enable natural interruption handling."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}