Today we release IFStruct, a new benchmark to measure how well models generate structured outputs.
A 350M model trained on it outperforms models more than 10x its size.
🧵
Structured output is one of the most common things we ask models to do and still where they break.
Most benchmarks test with clean, finalized schema. Real requests use plain language, paste an annotated example, switch formats halfway, and slip in constraints like "no code fence" or "no commentary."
(2/n)
IFStruct presents requirements in all of those forms: chat requests, bullet lists with explicit paths, raw JSON Schema, annotated JSON or YAML, ASCII tables. Half are rewritten into natural prose. Scoring is binary. Every field, type, enum, bound, and count right, with no invented keys.
The same generator that builds the eval builds training data just as easily. The same yes/no check that scores the benchmark can train the model.
(3/n)
LFM2.5-350M starts at 21.10% and reaches 44.90% after training, ahead of Qwen3.5-4B at 36.25% and granite-4.0-h-tiny at 38.75%. Frontier models near 100%.
(4/n)
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
structured output — Forcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Why it matters
IFStruct scores structured outputForcing a model's response to match a schema, so downstream code can parse it instead of guessing at prose.Full definition → on messy real requests with a strict all-fields-correct check. Frontier models sit near 100% while small models trail badly, and the same generator that builds the evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → builds RL training data.
Read the full thread on X ↗
