Urgent.News

What's breaking now, across thousands of outlets.

AI

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.

Q&A: Enabling more human-like interactions and understanding in multimodal foundation models

Humans perceive and interact with the world through sensory information. This could involve learning associations between bird chirps and the word "robin," or visually identifying and tracking a friend named "Sarah" among a crowd of people. People often rely on their latent "multimodal" knowledge, drawing on information across various data types like audio, video, text, and spoken words, to perform these tasks with minimal examples.

Multimodal foundation models of artificial intelligence are becoming increasingly adept at mimicking this capability. However, these models' apparent fluency in handling multimodal information may not always reflect their true understanding. Researchers at MIT-IBM Computing Research Lab, co-led by Jim Glass, a senior research scientist, have been exploring the limitations of state-of-the-art multimodal models and developing methods to overcome these challenges using limited supervision.

In an interview, Glass discussed the project's goals and their approach to addressing these model shortcomings.

The team has been investigating the relationship between perception (visual and auditory inputs) and language, leveraging the knowledge and language skills of large language models to understand how these domains can be intertwined without relying on extensive annotated training data. By creating novel benchmarks for evaluation and developing techniques that improve performance on these benchmarks, the researchers aim to enable models to learn autonomously.

The project has evolved from building models capable of learning correspondence associations automatically – essentially teaching machines to read picture books to children learning language – to incorporating general audio and video elements. The researchers used techniques such as masking parts of the input data and forcing the model to reconstruct the missing information, enabling it to learn robust relationships between sensory data and language.

Recent highlights of the project include the IPLoc research, which focused on how state-of-the-art models like GPT-4 struggle with localizing specific objects based on limited examples. The team developed a technique using pseudonym substitution, where they replaced words like "airplane" with "Elizabeth," compelling the model to pay closer attention to visual context rather than relying on previously learned associations.

Another notable paper, OmniR1, examined audio-large language models and revealed that some benchmarks used for evaluation might not truly assess the model's understanding of the audio. Instead, models sometimes opted for simple strategies, like guessing answers based on the multiple-choice options rather than accurately interpreting the audio content.

To address this issue, the researchers proposed evaluating these models across a variety of benchmark types, beyond simple multiple-choice questions. They also explored feeding random noise to the models to test their ability to make accurate predictions, further refining their techniques to enhance model performance without the need for extensive annotated data.

Written by urgent.news from MIT News AI's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

This story

This is one outlet's version. Read the fullest account.

Every outlet that reported this, and when →

Read the original at news.mit.edu →

More in AI

Miami coach Mario Cristobal updates Malachi Toney's recovery, eyes possible return within 2 weeks

5 Miami could have star receiver Malachi Toney back from his surgically repaired high ankle sprain within two weeks, Hurricanes coach Mario Cristobal said Friday.That timetable — if it holds up…

  • Star receiver Malachi Toney may return in two weeks from high ankle sprain.
  • Coach Mario Cristobal optimistic about Toney's recovery progress.
  • Toney's return could enable Miami to play in ACC championship on Dec 5th.

More from Friday 20 November →