TutorMoments: Do AI tutors know when to help and when to hold back?

In a groundbreaking study called TutorMoments, researchers introduce a method to assess whether advanced language models can effectively balance the critical challenge of determining when to assist students in education and when to refrain from doing so. This evaluation is based on real one-on-one math tutoring sessions, where experienced math teachers review transcripts collected from a U.S. tutoring program and identify moments where the tutor must decide between simplifying the problem for the student or challenging them to continue reasoning independently.

By replaying these transcripts into a language model acting as the tutor, the study determines how well the AI performs in these simulated sessions. While models tend to over-assist by providing too much support and infrequently encouraging deeper thinking, specifying the trade-off between helping and holding back in the model's prompt can improve performance. However, it falls short of matching the precision of human tutors, and language models continue to exhibit significant variability in making these judgments.

To promote open research, the study releases a dataset of de-identified tutoring transcripts, the code for the replay pipeline, and model replay recordings of key moments from those transcripts. The goal is to empower educators, researchers, and AI developers to better understand how language models handle pedagogical decisions that are crucial for effective learning, with the ultimate aim of creating tutors that adapt to each student's needs rather than completing the work for them.