**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**

In conversations, two types of information are conveyed: the content spoken and the speaker who conveyed it. Speech recognition transcribes the spoken words, while speaker diarization identifies who spoke when. Without this distinction, transcripts lack the ability to attribute commitments, objections, and interruptions to specific participants.

This limits search capabilities, summaries, action items, conversation analytics, and voice-agent memory. Speaker diarization detects active speech intervals, even in overlapping speech scenarios, which are then combined with automatic speech recognition (ASR) to generate a speaker-attributed transcript. NVIDIA Nemotron 3 Diarization is an open-weight model with 100 million parameters that ranks first on VoiceArena's Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER).

It supports up to eight speakers and handles overlapping speech, chunked processing for variable recording lengths, and customizable streaming latency. Earlier models like NVIDIA Streaming Sortformer were designed for four-speaker diarization, with Nemotron 3 Diarization extending support to eight speakers and improving accuracy and throughput.

Diarization involves detecting speech and assigning it to the correct speaker while preserving that assignment throughout the conversation, even during silences, interruptions, or long gaps. Streaming systems face challenges in maintaining accurate assignments due to limited context. Nemotron 3 Diarization addresses this by ordering output speakers based on their first appearance, creating stable speaker labels without the need for frequent permutation solving.

The model was trained on public and licensed speech data, including annotated and real-world conversations. Adding David AI data reduced the compound diarization error rate by 0.77 points. Nemotron 3 Diarization processes 16 kHz, single-channel audio into Mel-spectrogram features, arranging them in a tensor format with eight possible speaker channels.

Postprocessing converts these probabilities into generic speaker labels with start and end timestamps. During streaming inference, memory mechanisms like input buffers and queues help maintain context. This model can be integrated into a speaker-attributed transcription pipeline, combining diarization timestamps with ASR output to map spoken words to specific speakers.