The Parser Already Knows: Lightweight Bias Correction in Constrained Decoding
Source
arxiv.org
Author
I\c{s}{\i}l \"Ozg\"u, Yaoxuan Wu, Guy Van den Broeck, Miryung Kim
Date
Context
Forcing a LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → to obey a grammar can change which valid outputs it prefers. This paper proposes correcting that bias using information the parser already tracks about the developing text, together with possible next tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. A small correction component is trained offline and adjusts generation scores while leaving the base model's weights unchanged. Research abstract.
Across several grammars, the authors report bringing the constrained output distribution closer to the model's original distribution. The contribution is a way to preserve more of the model's preferences while enforcing the required syntax.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
articleVoice Memory for Agentic Speech RecognitionChao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg
articleA Closed-Loop Control Architecture for Reliable Constraint Satisfaction in LLM Text GenerationQuan Zhou, Shahbaz Siddeeq, Mika Saari, Pekka Abrahamsson
articleCompound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and UrgencyShrenik Jadhav, Nickalsa LaPlaca, Caleb Stone, Ashok Raja, Omar Ochoa, Vidhyashree Nagaraju