Implementing Deep Reinforcement Learning Models with Tensorflow + OpenAI Gym

A deep Q-network learns to estimate the future reward from taking an action, but its training target also depends on a neural-network estimate. If both change together, learning can become unstable. Lilian Weng's 2018 implementation tutorial shows how a separate target network helps steady that calculation.

The main network is trained against the immediate reward plus a future-value estimate from the target network. That target stays fixed during the main network's update, then receives periodic copies or gradual updates of its parameters. This makes concrete the idea of learning from existing value estimates, with a mechanism for slowing changes to those estimates.

“The OpenAI Gym toolkit provides a set of physical simulation environments, games, and robot simulators that we can play with and design reinforcement learning agents for.”

Lilian Weng

“Deep Q-network is a seminal piece of work to make the training of Q-learning more stable and more data-efficient, when the Q value is approximated with a nonlinear function. Two key ingredients are experience replay and a separately updated target network.”

Lilian Weng

“This two-step reinforcing procedure could potentially lead to overestimation of an (already) overestimated value, further leading to training instability.”

Lilian Weng

“The dueling Q-network ( Wang et al., 2016 ) is equipped with an enhanced network architecture: the output layer branches out into two heads, one for predicting state value, V, and the other for advantage , A.”

Lilian Weng

“The actor-critic algorithm learns two models at the same time, the actor for learning the best policy and the critic for estimating the state value.”

Lilian Weng