Exploration Strategies in Deep Reinforcement Learning

Rewarding an agent for surprise can encourage exploration, but unpredictable noise can keep paying that reward without teaching anything useful. Lilian Weng's 2020 survey illustrates this with the noisy-TV thought experiment: a screen showing random images continually defeats prediction and can distract an agent from its task.

One alternative described in the survey rewards learning progress, measured by a decrease in prediction error over time. A difficult situation becomes attractive when the agent is getting better at predicting it. This aims to distinguish persistent unpredictability from something learnable, explaining why the definition of an exploration bonus matters as much as the decision to reward exploration at all.

“Modern RL algorithms that optimize for the best returns can achieve good exploitation quite efficiently, while exploration remains more like an open topic.”