AI-assisted learning platform helps robots reason beyond literal commands

A new AI-assisted learning platform developed by MIT engineers, named FLEX, is enabling robots to understand the broader meaning behind human natural language commands. This innovative system, introduced in a study to be presented at the Conference on Robot Learning, allows humans to interact with robots using conversational language, and the robot can follow the instructions literally while also interpreting the words to draw generalizations that improve its motion-planning task.

FLEX, which stands for flexible feature-editing pipeline for robot motion-planning, incorporates a large language model (LLM) to recognize patterns, trends, and generalizations from vast amounts of text data found on the internet, such as social media posts, video transcriptions, and web content. This enables the robot to reason why it should avoid potential hazards, even if they were not explicitly mentioned in the command.

For instance, if a person instructs a robot to serve a cup of coffee while avoiding spilling it on a laptop, the robot uses the LLM to understand that liquids can damage electronics, and thus avoids bumping into other electronic devices as well.

The three-step process of FLEX is designed to help robots absorb more information from natural language commands, beyond what is explicitly stated. The first step, "Add," instructs the robot to add a feature to its environmental list based on the user's command. The second step, "Generalize," uses the LLM to identify patterns and connections among the specified features, allowing the robot to make generalizations about what to avoid.

Finally, the third step, "Mask," helps the robot update its understanding of the environment as objects are removed or no longer relevant.

This innovative learning platform has been tested in simulated scenarios, such as a robot carrying a cup of coffee on a cluttered table, where the person might instruct it to avoid spilling on a laptop. FLEX enables the robot to reason about the potential consequences of its actions, adjusting its path to avoid both the laptop and nearby electronic devices without the person having to explicitly specify all potential hazards.

The goal of FLEX is to allow robots to understand and respond to human commands more intuitively, helping to advance the development of assistive and household robots capable of performing physical tasks while reasoning and adapting in real-time.