Vibeleaderboard
Index / tool
Visit github.com
Category
Developer Tools
Rank
No. 1155Tools index

Previous survey · No. 1109 ·

Pricing
Open Source
Type
TOOL
Use case
Model & Agent Evaluation
Date

About

An OpenAI benchmark measuring how well agents do machine-learning engineering, using 75 Kaggle competitions as the task set and Kaggle's own medal thresholds as the bar. Agents get a fixed budget — 24 hours, 36 vCPUs, 440GB RAM and a single 24GB A10 — and are scored on the share of competitions where they take any medal, run over at least three seeds and reported with standard error. A 22-competition low-complexity split exists for cheaper runs.

What it does

MLE-bench turns real Kaggle contests into offline tasks for AI agents. A setup command pulls each contest's data through the Kaggle API, carves out a private test split with hidden answers, and leaves the agent a public folder plus written instructions. The agent trains a model inside a Docker container and must produce one CSV file. Grading compares that file with the hidden answers, then places the result against a saved copy of the original human standings, so each attempt reads as gold, silver, bronze or no medal. The package ships the data builders, the per-contest graders and the harness that ran the original agents.

Why it's ranked here

This is OpenAI's own yardstick for agents doing applied machine learning work, and the code backs the claim: every grader, every data split recipe and the agent harness are public under MIT, so a published number can be rerun rather than taken on trust. Results are anchored to real human Kaggle standings instead of an invented scale. The weak spots are cost and openness to new entrants. The full set needs 3.3TB of data and up to a day of GPU time per task, and the maintainers have paused new public submissions while they rework how fairness is checked.

What's good

Answers are kept away from the agent by construction. It sees a read-only public folder, the hidden answers sit in a separate read-only mount, and the agent process runs as a non-root user. A checker inside the container says whether a submission file is well formed without revealing any score, so an agent cannot tune against the test split. Downloaded archives are checked against stored fingerprints before anything is prepared. The Lite split cuts the job to 22 contests and about 158GB of data while still lining up with one column of the published results.

Tradeoffs

Setup is heavy. You need a Kaggle account, Git LFS and Docker, and you must accept each contest's rules on Kaggle; the setup step stops and asks you to open a browser when you have not. The dependency list pulls in TensorFlow, a MongoDB client and computer vision tooling for graders you may never run. The MIT grant covers the code, not the downloaded datasets. The bundled run script defaults to one seed although the authors recommend at least three. Two extra submissions were set aside for using test-set feedback, a reminder that results depend on honest harnesses.

How to use it well

Use it to build or compare agents that write and train models end to end, not to judge chat quality or code completion. Start with the included dummy agent on a single small contest to confirm the container, the submission checker and the private-data lockout work on your machine. Move to Lite before the full 75. Run three or more seeds, report the mean with standard error as the authors ask, and say so when you change the 24-hour, 36-CPU, single A10 budget. Run untrusted agents under the sysbox runtime rather than plain Docker.

Technical notes+

pyproject.toml declares package mlebench 1.0.0 (Python >=3.11) with a mlebench console script pointing at mlebench/cli.py, whose subcommands are prepare (single id, --lite, --all or --list), grade (a JSONL of submissions), grade-sample (one CSV) and dev download-leaderboard. mlebench/registry.py builds each Competition from a per-competition config under the package's competitions directory and resolves raw, prepared/public and prepared/private directories beneath a user cache data dir. mlebench/data.py downloads via the Kaggle client with a tenacity retry (3 attempts, 5s apart), prompts interactively to accept rules, MD5-checks the zip and the prepared files against stored checksums, and calls each competition's prepare function with raw, public and private arguments. mlebench/grade.py grade_csv scores a CSV with the competition grader, ranks it against the stored Kaggle leaderboard to set gold, silver, bronze and median thresholds, and marks any_medal; aggregate_reports sums medal counts across reports. environment/grading_server.py is a Flask app on port 5000 whose /validate route runs validate_submission and returns only a validity message. agents/run.py mounts public data read-only at /home/data and private data read-only under /private/data, waits up to 60s for /health, then execs the agent start script as user nonroot. run_agent.py fans tasks (competition x seed) over asyncio workers, defaults --n-seeds and --n-workers to 1, and refuses privileged agents unless I_ACCEPT_RUNNING_PRIVILEGED_CONTAINERS is set. environment/instructions.txt is the prompt handed to agents and fixes the submission path at /home/submission/submission.csv.

Observed

License
MIT for the code; the LICENSE note excludes external datasets fetched at use
Language
Python, requires 3.11 or newer
Packaging
setuptools package mlebench with an mlebench console command
Interfaces
CLI with prepare, grade, grade-sample and dev download-leaderboard; separate agent run script
Data source
Kaggle API, client pinned below 1.7; per-competition rule acceptance required
Runtime
Docker, sysbox runtime recommended, NVIDIA Container Toolkit optional for GPUs
In-container checker
Local HTTP endpoint reports submission validity only, never a score
Task set
75 competitions; Lite split of 22 low-complexity competitions
Data volume
About 158GB for Lite versus 3.3TB for the full set

Read from README.md, pyproject.toml, LICENSE, mlebench/cli.py, mlebench/grade.py, mlebench/registry.py, mlebench/data.py, environment/grading_server.py, environment/instructions.txt, agents/README.md, agents/run.py, run_agent.py.

Intel on MLE-bench

More in Intel

Tags

benchmarkml-engineeringagentskaggleevaluationopenai

Tech Stack

Python

Comments (0)

No comments yet

Editorially curated, with community endorsements as a secondary signal. Corrections welcome.