Policy Gradient Algorithms

A reinforcement-learning policy assigns probabilities to actions. Updating those probabilities too aggressively can undo useful behavior. Lilian Weng's policy-gradient survey presents PPO as one approach to making those updates more restrained.

PPO, from the original paper, optimizes a surrogate objective over multiple epochs of minibatch updates. As OpenAI's Spinning Up explanation describes, it compares each sampled action's probability under the new and previous policies. For an action with a positive advantage, the objective stops rising once the ratio passes a chosen threshold, so going further earns nothing. Clipping changes the training objective; it does not impose a hard limit on every resulting action probability.

“The goal of reinforcement learning is to find an optimal behavior strategy for the agent to obtain optimal rewards. The policy gradient methods target at modeling and optimizing the policy directly.”

Lilian Weng

“It is natural to expect policy-based methods are more useful in the continuous space. Because there is an infinite number of actions and (or) states to estimate the values for and hence value-based approaches are way too expensive computationally in the continuous space.”

Lilian Weng

“The policy gradient theorem lays the theoretical foundation for various policy gradient algorithms. This vanilla policy gradient update has no bias but high variance. Many following algorithms were proposed to reduce the variance while keeping the bias unchanged.”

Lilian Weng

“The off-policy approach does not require full trajectories and can reuse any past episodes ( “experience replay” ) for much better sample efficiency.”

Lilian Weng

“To resolve the inconsistency, a coordinator in A2C waits for all the parallel actors to finish their work before updating the global parameters and then in the next iteration parallel actors starts from the same policy.”

Lilian Weng