TutorMoments: Do AI tutors know when to help and when to hold back?

TutorMoments tests whether an AI tutor chooses the right amount of help. Ai2's framework pauses real math-tutoring transcripts at teacher-identified decision points, then lets a model continue with a simulated student. Its scoring checks whether the tutor supplies needed support or asks the student to do more reasoning. The preview explains the replay method.

Every tested model scored better when its instructions explicitly described that trade-off than when simply told to tutor well. The result shows how strongly tutor behavior depends on its instructions. Because the continuations use simulated students and the dataset emphasizes missed teaching opportunities, these scores measure response choices rather than real student learning or overall superiority to human tutors.