Lilian Weng

Vision-language systems traditionally bolt an object detector onto a text decoder. A look at how generalized vision-language models replace that pipeline, for captioning and visual question answering.

[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.] [Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.] [Updated on 2021-09-19: Add “unlikelihood training” .]

[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.] [Updated on 2018-11-06: Add a link to the implementation of Transformer model.] [Updated on 2018-11-18: Add Neural Turing Machines .] [Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.] [Updated on 2020-04-07: A follow-up post on improved Transformer models is here .]

[Updated on 2020-02-03: mentioning PCG in the “Task-Specific Curriculum” section. [Updated on 2020-02-04: Add a new “curriculum through distillation” section.

Deep RL is too sample-hungry to train on real robots, so models are trained in simulation and fail on the reality gap. Domain randomization closes it by varying the simulator enough that reality looks like one more variation.

A survey of policy gradient algorithms in reinforcement learning, from the basic theorem through actor-critic variants including SAC, D4PG, TD3 and SVPG.

[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)]. [Updated on 2022-08-27: Added classifier-free guidance , GLIDE , unCLIP and Imagen . [Updated on 2022-08-31: Added latent diffusion model . [Updated on 2024-04-13: Added progressive distillation , consistency models , and the Model Architecture section .

Special thanks to John Schulman for a lot of super valuable feedback and direct edits on this post. Test time compute ( Graves et al. 2016 , Ling, et al. 2017 , Cobbe et al. 2021 ) and Chain-of-thought (CoT) ( Wei et al. 2022 , Nye et al. 2021 ), have led to significant improvements in model performance, while raising many research questions. This post aims to review recent developments in how to effectively use test-time compute (i.e. “thinking time”) and why it helps.

Human vocabulary comes in free text. In order to make a machine learning model understand and process the natural language, we need to transform the free-text words into numeric values. One of the simplest transformation approaches is to do a one-hot encoding in which each distinct word stands for one dimension of the resulting vector and a binary value indicates whether the word presents (1) or not (0).

Recursive self-improvement dates to I. J. Good in 1965 and Yudkowsky's 2008 framing of the feedback loop. This traces the idea forward to harness engineering, where the improvement happens around the model rather than inside it.

Exploration versus exploitation in deep reinforcement learning: why committing to a solution too early risks local minima, and the strategies including curiosity and exploration via disagreement that push agents to keep looking.

The exploration-exploitation dilemma stated as the multi-armed bandit problem, with the standard family of solutions from epsilon-greedy through UCB and Thompson sampling, implemented for Bernoulli bandits.

In my earlier post on meta-learning , the problem is mainly defined in the context of few-shot classification. Here I would like to explore more into cases when we try to “meta-learn” Reinforcement Learning (RL) tasks by developing an agent that can solve unseen tasks fast and efficiently.

A walk from the original generative adversarial network to Wasserstein GAN, covering why the game-theoretic setup of generator against critic is unstable and what the Wasserstein distance fixes.

Narrowing hallucination to its useful meaning: output that is fabricated and grounded in neither the provided context nor world knowledge, rather than any mistake a model makes.

Scaling laws say training loss falls predictably as model size, data and compute grow, a straight line on a log-log plot. A careful look at what that framework does and does not license you to conclude.

Although most popular and successful model architectures are designed by human experts, it doesn’t mean we have explored the entire network architecture space and settled down with the best option. We would have a better chance to find the optimal solution if we adopt a systematic and automatic way of learning high-performance model architectures.