Neural tracking of surprisal and semantic distance in naturalistic movie viewing
Understanding speech requires listeners to integrate incoming input with prior linguistic and thematic knowledge to access meaning, a task greatly aided by prediction. Surprisal and related phenomena (e.g., next word prediction) tend to be associated with broad activation of language regions during listening. A major challenge for interpreting the neural correlates of surprisal is that lexical…
Comprehending speech involves listeners combining incoming auditory and visual input with their existing knowledge to derive meaning. This process is bolstered by prediction, where surprisal and semantic distance are key factors. When predicting the next word, surprisal and related phenomena, such as next-word prediction, typically lead to activation across language regions. However, semantic distance provides a more straightforward measure of how related two language elements are in a high-dimensional semantic space.
The study examined both surprisal and semantic distance in naturalistic discourse during the viewing of the full-length film 500 Days of Summer. Researchers analyzed fMRI scans from 20 English-speaking adults as they watched the film word-by-word. The focus was on neural sensitivity to either surprisal or semantic distance. By utilizing hierarchical voxelwise encoding models and controlling for low-level auditory and visual features, the addition of either surprisal or semantic distance enhanced prediction in several brain areas.
Specifically, both surprisal and semantic distance improved prediction signals in bilateral superior temporal gyrus, left inferior frontal gyrus, and right cerebellum. Crucially, the two factors explained unique variations in bilateral temporal cortex, even when accounting for each other. These findings support the idea that the brain simultaneously tracks probabilistic prediction and semantic integration.
More broadly, the results show that surprisal and semantic distance can predict meaningful neural variations even within a complex audiovisual context.
Written by urgent.news from bioRxiv's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
