System helps humans predict when self-driving cars will make mistakes

Self-driving cars often rely on deep learning models that can sometimes make unexpected mistakes. For example, a self-driving car might suddenly brake and block the path of an emergency vehicle. In these situations, human drivers or passengers may need to react quickly to avoid a collision. Researchers at MIT and Motional have developed a new method to help humans anticipate these mistakes more effectively.

The method, called Concept-Wrapper Network (CW-Net), provides clear explanations of the machine learning model's decision-making process.

Typically, the internal reasoning of deep learning models is difficult to understand. CW-Net addresses this issue by translating the complex decision-making process into understandable concepts. These concepts, such as "approaching stopped vehicle" or "close to cyclist," can help drivers and passengers better understand the vehicle's behavior, improving their situational awareness. CW-Net also ensures that the explanations are accurate and do not mislead users.

In road tests on a private track, CW-Net explanations helped safety drivers more accurately predict vehicle behavior, and a larger simulation study with non-expert users yielded similar results. This technique could significantly enhance the safety and transparency of autonomous vehicles, building the necessary trust for their use.

As Julie Shah, an MIT professor, explains, "Unless we are building these technologies in a way that we can rely on and predict their behavior, then it is a shaky and unsafe foundation for their use."

The research, led by Eoin Kenny, a former MIT postdoc and senior AI researcher at J.P. Morgan Chase, along with co-senior author Momchil Tomov from Motional, is published in Nature. CW-Net works by acting as a "concept classifier" within an autonomous vehicle's machine-learning planner architecture. It translates the model's internal reasoning process into understandable concepts while ensuring those explanations accurately reflect the true reasons behind the vehicle's behavior.