MLE-bench
github.com/openai/mle-bench- Category
- Developer Tools
- Rank
- No. 1155Tools index
Previous survey · No. 1109 ·
- Listed in
- Find AI benchmarks
- Pricing
- Open Source
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 1.8k stars
- Date
About
An OpenAI benchmark measuring how well agents do machine-learning engineering, using 75 Kaggle competitions as the task set and Kaggle's own medal thresholds as the bar. Agents get a fixed budget — 24 hours, 36 vCPUs, 440GB RAM and a single 24GB A10 — and are scored on the share of competitions where they take any medal, run over at least three seeds and reported with standard error. A 22-competition low-complexity split exists for cheaper runs.
What it does
MLE-bench turns real Kaggle contests into offline tasks for AI agents. A setup command pulls each contest's data through the Kaggle API, carves out a private test split with hidden answers, and leaves the agent a public folder plus written instructions. The agent trains a model inside a Docker container and must produce one CSV file. Grading compares that file with the hidden answers, then places the result against a saved copy of the original human standings, so each attempt reads as gold, silver, bronze or no medal. The package ships the data builders, the per-contest graders and the harness that ran the original agents.
Why it's ranked here
This is OpenAI's own yardstick for agents doing applied machine learning work, and the code backs the claim: every grader, every data split recipe and the agent harness are public under MIT, so a published number can be rerun rather than taken on trust. Results are anchored to real human Kaggle standings instead of an invented scale. The weak spots are cost and openness to new entrants. The full set needs 3.3TB of data and up to a day of GPU time per task, and the maintainers have paused new public submissions while they rework how fairness is checked.
What's good
Answers are kept away from the agent by construction. It sees a read-only public folder, the hidden answers sit in a separate read-only mount, and the agent process runs as a non-root user. A checker inside the container says whether a submission file is well formed without revealing any score, so an agent cannot tune against the test split. Downloaded archives are checked against stored fingerprints before anything is prepared. The Lite split cuts the job to 22 contests and about 158GB of data while still lining up with one column of the published results.
Tradeoffs
Setup is heavy. You need a Kaggle account, Git LFS and Docker, and you must accept each contest's rules on Kaggle; the setup step stops and asks you to open a browser when you have not. The dependency list pulls in TensorFlow, a MongoDB client and computer vision tooling for graders you may never run. The MIT grant covers the code, not the downloaded datasets. The bundled run script defaults to one seed although the authors recommend at least three. Two extra submissions were set aside for using test-set feedback, a reminder that results depend on honest harnesses.
How to use it well
Use it to build or compare agents that write and train models end to end, not to judge chat quality or code completion. Start with the included dummy agent on a single small contest to confirm the container, the submission checker and the private-data lockout work on your machine. Move to Lite before the full 75. Run three or more seeds, report the mean with standard error as the authors ask, and say so when you change the 24-hour, 36-CPU, single A10 budget. Run untrusted agents under the sysbox runtime rather than plain Docker.
Technical notes+
pyproject.toml declares package mlebench 1.0.0 (Python >=3.11) with a mlebench console script pointing at mlebench/cli.py, whose subcommands are prepare (single id, --lite, --all or --list), grade (a JSONL of submissions), grade-sample (one CSV) and dev download-leaderboard. mlebench/registry.py builds each Competition from a per-competition config under the package's competitions directory and resolves raw, prepared/public and prepared/private directories beneath a user cache data dir. mlebench/data.py downloads via the Kaggle client with a tenacity retry (3 attempts, 5s apart), prompts interactively to accept rules, MD5-checks the zip and the prepared files against stored checksums, and calls each competition's prepare function with raw, public and private arguments. mlebench/grade.py grade_csv scores a CSV with the competition grader, ranks it against the stored Kaggle leaderboard to set gold, silver, bronze and median thresholds, and marks any_medal; aggregate_reports sums medal counts across reports. environment/grading_server.py is a Flask app on port 5000 whose /validate route runs validate_submission and returns only a validity message. agents/run.py mounts public data read-only at /home/data and private data read-only under /private/data, waits up to 60s for /health, then execs the agent start script as user nonroot. run_agent.py fans tasks (competition x seed) over asyncio workers, defaults --n-seeds and --n-workers to 1, and refuses privileged agents unless I_ACCEPT_RUNNING_PRIVILEGED_CONTAINERS is set. environment/instructions.txt is the prompt handed to agents and fixes the submission path at /home/submission/submission.csv.
Observed
- License
- MIT for the code; the LICENSE note excludes external datasets fetched at use
- Language
- Python, requires 3.11 or newer
- Packaging
- setuptools package mlebench with an mlebench console command
- Interfaces
- CLI with prepare, grade, grade-sample and dev download-leaderboard; separate agent run script
- Data source
- Kaggle API, client pinned below 1.7; per-competition rule acceptance required
- Runtime
- Docker, sysbox runtime recommended, NVIDIA Container Toolkit optional for GPUs
- In-container checker
- Local HTTP endpoint reports submission validity only, never a score
- Task set
- 75 competitions; Lite split of 22 low-complexity competitions
- Data volume
- About 158GB for Lite versus 3.3TB for the full set
Read from README.md, pyproject.toml, LICENSE, mlebench/cli.py, mlebench/grade.py, mlebench/registry.py, mlebench/data.py, environment/grading_server.py, environment/instructions.txt, agents/README.md, agents/run.py, run_agent.py.
Intel on MLE-bench
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.