The authors trace uneven RL results to two weak points. First, hard queries with high semantic entropy often yield sample groups that are all wrong, collapsing the group-relative advantage to zero exactly where hallucinationWhen a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.Full definition → risk is highest.
Second, confident-but-wrong tokens barely get corrected: a categorical policy's expected score-gradient norm vanishes as its distribution sharpens, so the predictions most in need of correction receive the weakest updates.
DEEPO adds expert prefixes with groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → continuations on high-uncertainty queries to restore advantage variance, and applies advantage-sign-aware Renyi preconditioning so correction reaches confident errors.
Each branch improves on GRPO alone. Their interaction is statistically significant on VideoMMMU (+4.0, 95% CI 1.1 to 6.9) and additive on the other tasks.
Terms in this piece · Glossary
hallucination — When a model states something false with full confidence — inventing facts, citations, or APIs that don't exist.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
Why it matters
Explains a specific mechanical reason RL fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → often fails to reduce hallucination on the hardest, highest-risk queries, and offers a targeted fix instead of just more RL compute.
articleOpen challenges in LLM researchChip Huyen
articleUnified Hallucination Fuzzing for Multimodal Large Language ModelsPengfei Zhou, Jiajun Song, Zhiwei Tang, Yixing Ma, Xiaopeng Peng, Donghui Si, Yuhang Xu, Huiqi Song, Yiyuan Miao, Yichen Qian, Weihua Chen, Wangbo Zhao, Bohan Zhuang, Jiasheng Tang, Yang You