A comprehensive, math-first survey of policy gradient methods that walks through the derivations and design tradeoffs of REINFORCE, A2C/A3C, DDPG, TD3, SAC, PPO, IMPALA and more in one place — a solid reference for anyone building or tuning RL-based systems.
Key quotes
“The goal of reinforcement learning is to find an optimal behavior strategy for the agent to obtain optimal rewards. The policy gradient methods target at modeling and optimizing the policy directly.”
Lilian Weng
“It is natural to expect policy-based methods are more useful in the continuous space. Because there is an infinite number of actions and (or) states to estimate the values for and hence value-based approaches are way too expensive computationally in the continuous space.”
Lilian Weng
“The policy gradient theorem lays the theoretical foundation for various policy gradient algorithms. This vanilla policy gradient update has no bias but high variance. Many following algorithms were proposed to reduce the variance while keeping the bias unchanged.”
Lilian Weng
“The off-policy approach does not require full trajectories and can reuse any past episodes ( “experience replay” ) for much better sample efficiency.”
Lilian Weng
“To resolve the inconsistency, a coordinator in A2C waits for all the parallel actors to finish their work before updating the global parameters and then in the next iteration parallel actors starts from the same policy.”
