A benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → that separates visual ability from language ability shows frontier multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → models failing tasks that 3-year-olds solve. It indicates where vision-based agents are unreliable, so you can avoid relying on them there.
Key takeaways · AI-distilled
On the full 388-question BabyVision set, adults scored 94.1%; the best model, Gemini3-Pro-Preview, reached 49.7%, GPT-5.2 34.4%, Claude-4.5-Opus 14.2%, and the best open model, Qwen3VL-235B-Thinking, 22.2%.
The authors call these tasks "unspeakable": models describe shapes, curves or 3D scenes in coarse text before reasoning, which loses fine detail, makes them switch tracks at line crossings, and leads to miscounted hidden blocks.
In a preliminary study, GRPO fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → of Qwen3-VL-8B-Thinking on 1,400 examples for 450 steps raised BabyVision accuracy from 13.1% to 17.9%, but visual tracking fell 1.2 points; the authors hypothesize it is the least amenable to verbalization.
BabyVision-Gen (280 questions, auto-grader with 0.96 human agreement) scored Nano-Banana-Pro at 18.3%, GPT-1.5-Image at 9.8% and Qwen-Image-Edit-2511 at 4.8%; the authors see drawing-based reasoning as promising but rarely fully correct today.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
