BabyVision
github.com/unipat-ai/babyvision- Category
- Developer Tools
- Rank
- No. 1421Tools index
Previous survey · No. 1374 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Interfaces
- CLI
- GitHub
- 265 stars
- Date
About
Evaluates visual reasoning through discrimination, tracking, spatial perception, and pattern recognition. Tool-assisted results use a different setup from direct vision answers.
What it does
BabyVision is a set of picture puzzles that small children solve easily and multimodal models mostly fail: spot the odd one out, trace a maze, fold a paper net into a cube, finish a pattern. It ships two runners. One sends each puzzle image and question to a vision-language model, then has a second language model grade the final answer against the key. The other asks image generators to draw the answer onto the puzzle, circling or tracing it, and has a vision model compare that drawing with a reference answer image.
Why it's ranked here
It measures a gap that text-heavy tests hide. The authors report humans at 94.1 percent, the strongest model they tested, Gemini 3 Pro Preview, at 49.7, and Claude 4.5 Opus at 14.2. The drawing track is harsher: the published summary has the best image model near 18 percent across three runs. The harness backs those numbers with repeat passes and a spread rather than one run. Against that, the license is a research-only statement rather than a named license, and every score depends on a language model acting as grader.
What's good
The scoring is honest about noise. Each model runs three passes by default, and results come back as a mean with a standard deviation, broken down by the four puzzle families and their subtypes. The image grader gets instructions written per puzzle type, so a maze is judged on whether the path takes the same route and a counting task on whether the exact number matches. Missing drawings count as wrong instead of quietly leaving the total. The dependencies are six common Python packages, and any endpoint that speaks the OpenAI chat format works.
Tradeoffs
Correctness is decided by a language model, not exact matching, so results can move with the choice of judge; the defaults use GPT-5.2 for answers and Gemini 3 Flash Preview for drawings. The answer runner has a retry bug: when a puzzle fails all three attempts, the code crashes instead of recording a miss, which can end the whole pass with nothing saved. It also switches on a reasoning option specific to OpenRouter. The license line says only that the project is released for research purposes, which gives commercial users nothing firm to rely on.
How to use it well
Reach for it when you need to know whether a vision model can actually look rather than read: vetting a new multimodal model before trusting it with diagrams, maps or layout work. Run all three passes, report the spread, and keep the judge model fixed across every model you compare so the numbers stay comparable. Point it at a stable provider, since one persistent API failure can stop a run. It is a set of drawn puzzles, so it says little about photos or scanned documents, and the research-only terms need clearing before commercial use.
Technical notes+
babyvision_eval/evaluate_model.py loads meta_data.jsonl, base64-encodes each image via image_to_base64 in babyvision_eval/utils.py (PNG and JPG only, anything else raises), appends an instruction to answer in \boxed{}, and calls the model through the openai client with extra_body reasoning enabled, a 1800 second timeout and a multiprocessing Pool of 8 workers. extract_boxed_answer takes the last \boxed{} match (falling back to begin_of_box tokens) and LLM_JUDGE_PROMPT asks the judge for True or False; the parse is a substring test for 'true', and anything unparseable counts as False. The retry loop in process() runs 3 times, but the fallback return reads str(e) after the except block, where Python 3 has already unbound e, so an item that exhausts its retries raises UnboundLocalError, which pool.imap re-raises in the parent and the pass's JSON is never written. babyvision_eval/compute_score.py aggregates per file into overall, Type and Subtype accuracy and reports numpy mean and population std across passes. babyvision_gen_eval/scripts/evaluate.py sends input, ground-truth and generated images plus per-subtype criteria from get_type_criteria to an OpenRouter chat endpoint at temperature 0.0 with max_tokens 200, honours Retry-After on 429, resumes from existing eval.jsonl rows, and counts missing generations as incorrect; babyvision_gen_eval/results/summary.txt records 280 tasks and three rounds per model.
Observed
- License
- README states release for research purposes; no named open-source license
- Language
- Python
- Interface
- Command-line scripts with bash wrappers; no library or service API
- Model access
- Any OpenAI-compatible chat endpoint; defaults point at OpenRouter
- Grading method
- LLM judge for both tracks: text answers and generated annotation images
- Puzzle families
- Fine-grained discrimination, visual tracking, spatial perception, visual pattern recognition
- Generation track size
- 280 tasks, per the committed results summary
- Repeat runs
- Three passes by default, reported as mean and standard deviation
- Dependencies
- pillow, tqdm, requests, openai, regex, numpy (requirements.txt)
- Benchmark data
- Shipped as zip archives inside the repository
- Retry failure path
- An item that fails all retries raises instead of recording a miss (babyvision_eval/evaluate_model.py)
Read from README.md, requirements.txt, babyvision_eval/README.md, babyvision_eval/evaluate_model.py, babyvision_eval/compute_score.py, babyvision_eval/utils.py, babyvision_eval/run_inference.sh, babyvision_gen_eval/README.md, babyvision_gen_eval/scripts/evaluate.py, babyvision_gen_eval/results/summary.txt.
What it can do
Evaluate multimodal LLMs on visual reasoning benchmark tasks
Multimodal LLM → Performance score
Evaluate image generation models on visual reasoning tasks
Image generation model → Performance score
Intel on BabyVision
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.