Curriculum for Reinforcement Learning

A training curriculum can change as the learner improves. In Lilian Weng's 2020 survey, teacher-student methods assign one system to choose tasks for another. The teacher uses task scores to favor opportunities for learning progress or revisit skills at risk of being forgotten.

Selecting the next exercise becomes a decision under uncertainty, connecting curriculum design to the bandit problem. The goal is to spend training effort where it helps, but choosing tasks adaptively adds its own challenge: the survey reports that uniformly sampling tasks was a strong comparison in the discussed discrete-task studies. A curriculum earns its value through improved learning, not simply an easy-to-hard ordering.

“Humans differ from other species along many dimensions, but two are particularly noteworthy. Humans display an exceptional capacity to learn; and humans are remarkable for the unusually long time it takes to reach maturity.”

Jeffrey Elman

“Indeed, learning is probably the best superpower we humans have.”

Lilian Weng

“Keep in mind that, a bad curriculum may even hamper learning.”

Lilian Weng

“It is plausible that some curriculum strategies could be useless or even harmful.”

Lilian Weng