Towards general auditory intelligence for machine listening and speaking
Nature Machine Intelligence, Published online: 14 August 2026; doi:10.1038/s42256-026-01281-1 Wang et al. summarize advances in machine listening and speaking, speech-based interaction, and audio–visual understanding.
This Review presents a comprehensive look at recent advancements in incorporating audio into large language models, with a particular emphasis on four major domains: audio comprehension, audio generation, speech-based interaction, and audio-visual understanding. Researchers analyze how large language models are transforming audio perception and reasoning, allowing systems to grasp sound at a more profound semantic level, produce expressive audio outputs, and engage in human-like spoken exchanges.
Additionally, the fusion of audio and visual modalities is explored to enhance situational awareness and cross-modal reasoning, extending the frontiers of multimodal intelligence. Beyond summarizing existing research, the Review also highlights significant challenges and potential pathways for developing general auditory intelligence capable of perceiving, comprehending, and engaging with sound in a manner analogous to human cognition.
Written by urgent.news from Nature Machine Intelligence's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
