grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
A rigorous, well-organized primer on meta-learning that walks through metric-based, model-based, and optimization-based approaches (including MAML and memory-augmented networks) with the math and intuition — useful groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → for anyone building models that adapt quickly from few examples.
Key quotes
“A good machine learning model often requires training with a large number of samples. Humans, in contrast, learn new concepts and skills much faster and more efficiently. Kids who have seen cats and birds only a few times can quickly tell them apart.”
Lilian Weng
“We expect a good meta-learning model capable of well adapting or generalizing to new tasks and new environments that have never been encountered during training time. The adaptation process, essentially a mini learning session, happens during test but with a limited exposure to the new task configurations.”
Lilian Weng
“Meta-learning takes this idea one step further, rather than fine-tuning according to one down-steam task, it optimizes the model to be good at many, if not all.”
Lilian Weng
“This embedding method is called “Full Contextual Embeddings (FCE)”. Interestingly it does help improve the performance on a hard task (few-shot classification on mini ImageNet), but makes no difference on a simple task (Omniglot).”
Lilian Weng
“One faster way to learn is to utilize one neural network to predict the parameters of another neural network and the generated weights are called fast weights . In comparison, the ordinary SGD-based weights are named slow weights .”
articleMeta Reinforcement LearningLilian Weng
articleFrom User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking
articleMeasure, Don't Optimize: Forecasting Recovery in LLM UnlearningZirui Song, Huaxing Liu, Xiang Wang, Shuai Li, Xinye Li, Lang Gao, Jinghui Zhang, Zheng Lu, Fengxian Ji, Xiaojun Chang, Xiuying Chen