{
  "id": 20953974,
  "title": "Q&A: Enabling more human-like interactions and understanding in multimodal foundation models",
  "url": "https://urgent.news/2026/11/20/q-a-enabling-more-human-like-interactions-and-understanding-in",
  "topic": "science",
  "section": "Science",
  "published": "2026-11-20T21:00:00.000Z",
  "source": {
    "name": "MIT News Research",
    "slug": "mit-news-research",
    "url": "https://news.mit.edu/2026/qa-enabling-more-human-interactions-understanding-multimodal-foundation-models-1120"
  },
  "original_language": "en",
  "account": "The MIT-IBM Computing Research Lab team, led by Jim Glass, a senior research scientist, has been exploring the limitations of state-of-the-art multimodal AI models. Their aim is not to scale up these models but to address their weaknesses while minimizing the need for extensive annotated training data.\n\nInitially, the team focused on teaching AI to recognize correspondences between audio and visual data, akin to teaching a child to associate sounds with images. They demonstrated this using matching and mismatching examples. As they progressed, they expanded their scope to include general audio, followed by video sequences, and trained models to reconstruct and identify relationships between audio and visual data.\n\nThe team has also analyzed the performance of large multimodal models, revealing that even these advanced systems can struggle with tasks like object localization with minimal examples. To overcome this issue, they developed a technique involving pseudonym substitution, forcing the models to rely more on visual context rather than previously learned textual information.\n\nIn another study, the researchers examined audio-large language models, uncovering that some benchmarks used in evaluating these models are multiple-choice questions that may not accurately reflect the model's understanding of the audio. The models are sometimes merely guessing the answer based on pattern recognition rather than genuinely processing the audio content. To address this, the team advocates for evaluating models on a diverse array of benchmarks, including random noise inputs, to gauge their true understanding capabilities.",
  "summary": "Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}