Intel / article

Meta Reinforcement Learning

Source: lilianweng.github.io
Author: Lilian Weng
Date: Published Jun 23, 2019

Context

A reinforcement-learning can adapt its behavior without changing its neural-network weights. Lilian Weng's 2019 meta-RL survey describes recurrent agents trained across related tasks. Along with the current observation, they receive their previous action and reward; an internal memory accumulates evidence about the task they are facing.

During training, weight updates teach the agent how to use that history. On a new task, its changing memory can guide different decisions while those learned weights stay fixed. This is one way to implement learning to learn. The described evaluations use related task families, such as mazes with different layouts, which bounds what the adaptation demonstrates.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“In the modern days of DL, Wang et al. (2016) and Duan et al. (2017) simultaneously proposed the very similar idea of Meta-RL (it is called RL^2 in the second paper). A meta-RL model is trained over a distribution of MDPs, and at test time, it is able to learn to solve a new task quickly.”
Lilian Weng

“A recurrent neural network maintains a hidden state. Thus, it could acquire and memorize the knowledge about the current task by updating the hidden state during rollouts. Without memory, meta-RL would not work.”
Lilian Weng

“As a general ML rule, a learning algorithm with weak inductive bias will be able to master a wider range of variance, but usually, will be less sample-efficient. Therefore, to narrow down the hypotheses with stronger inductive biases help improve the learning speed.”
Lilian Weng

“Overall the capacity of episodic control is limited by the complexity of the environment. It is very rare for an agent to repeatedly visit exactly the same states in a real-world task, so properly encoding the states is critical.”
Lilian Weng

“POET ( Wang et al, 2019 ), a framework based on the evolutionary algorithm, attempts to generate tasks while the problems themselves are being solved. The implementation of POET is only specifically designed for a simple 2D bipedal walker environment but points out an interesting direction.”
Lilian Weng

Read the source lilianweng.github.io