When reward functions beat labeled data: reinforcement fine-tuning with GRPO
- Source
- x.com
- Date
New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with @Predibase, and taught by @TravisAddair, its Co-Founder and CTO, and @grg_arnav, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive…
Context
GRPO trains a model by generating several answers to the same prompt and comparing their rewards. The trainer documentation describes sampling a set of completions per prompt, computing each one's reward with a reward function or model, and calculating its advantage relative to the group.
The reinforcement fine-tuning course announcement presents rewards as a replacement for pre-existing training examples on tasks with verifiable outcomes, uses Wordle to teach reward design, and examines how models can exploit a poorly chosen scoring rule. This makes the ability to evaluate generated attempts central: teams still need training tasks and rewards that reflect the behavior they want.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
postSFT, DPO or online RL: choosing a post-training method by its dataAndrew Ng- articleGSPO: Towards Scalable Reinforcement Learning for Language ModelsQwen Team
articleGRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual SettingsKonstantin Dobler, Federico Scozzafava, Jonathan Janke, Mohamed Ali, Simon Lehnerer
Checking sign-in…
Loading comments…