IFStruct puts structured-output evals on messy, real-world schemas

Today we release IFStruct, a new benchmark to measure how well models generate structured outputs. A 350M model trained on it outperforms models more than 10x its size. 🧵

Structured output is one of the most common things we ask models to do and still where they break. Most benchmarks test with clean, finalized schema. Real requests use plain language, paste an annotated example, switch formats halfway, and slip in constraints like "no code fence" or "no commentary." (2/n)

IFStruct presents requirements in all of those forms: chat requests, bullet lists with explicit paths, raw JSON Schema, annotated JSON or YAML, ASCII tables. Half are rewritten into natural prose. Scoring is binary. Every field, type, enum, bound, and count right, with no invented keys. The same generator that builds the eval builds training data just as easily. The same yes/no check that scores the benchmark can train the model. (3/n)

LFM2.5-350M starts at 21.10% and reaches 44.90% after training, ahead of Qwen3.5-4B at 36.25% and granite-4.0-h-tiny at 38.75%. Frontier models near 100%. (4/n)