TutorMoments: Do AI tutors know when to help and when to hold back?

Ai2's TutorMoments examines whether an AI tutor knows when to offer help and when to leave more reasoning to the student. Teachers identify decision points in real math-tutoring transcripts. A model then continues each session with a simulated student, and an automated evaluator compares its behavior with the teachers' judgments. The preview report says explicit instructions about this trade-off improved every tested model's scores.

The result suggests that a generic instruction to be helpful is insufficient for this tutoring task. These scores measure behavior in simulated continuations; they do not measure whether real students learned more.