Measuring whether an AI tutor knows when to withhold the answer
- Source
- x.com
- Date
Today we're introducing a preview of TutorMoments, a framework that measures whether AI tutors can make one of the hardest calls in teaching: when to step in and help a student, & when to hold back and let them do the heavy thinking. 🧵
Language models are trained to be helpful, and a helpful assistant tends to do the hard part of learning for you: explains the concept, lays out the steps, & guides you to the answer. That can cut short the productive struggle that leads to stronger understanding.
TutorMoments is built on transcripts of real one-on-one math tutoring. We had experienced teachers read them & flag key moments—decision points where the tutor had to choose between making a problem easier & pushing the student to do more of the reasoning.
TutorMoments pauses a transcript at these key moments & lets an LLM take over as the tutor; another model stands in for the student. Each replay is scored: did the tutor support the student when needed, push for harder thinking when they were ready, & avoid over-helping?
Context
An AI tutor needs to judge when an explanation would help and when it would do the student's thinking for them. Ai2's TutorMoments tests that judgment by pausing real math-tutoring transcripts at teacher-selected decision points. A model continues as tutor, another plays the student, and an automated scorer checks whether the tutor's behavior fits teachers' judgments. How TutorMoments works.
All seven tested models scored better when explicitly prompted to balance assistance with intellectual challenge. Those scores measure behavior in simulated continuations, rather than learning by real children. The result gives tutor developers a concrete behavior to evaluate: whether help preserves an appropriate amount of work for the learner. Results and limitations.
Checking sign-in…
Loading comments…