BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
Source
arxiv.org
Author
John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu
Date
Why it matters
Coding benchmarks are saturating; this gives a structural difficulty predictor and a way to generate harder repo-level debugging tasks for evaluating ML engineering agents.
Key takeaways · AI-distilled
The Code Complexity Index scores tasks on ten structural features, including repository scale, dependency depth and patch size. It predicted average solve rates across eight public coding benchmarks with R² = 0.951, with patch size and complexity as the top predictors.
BuggyMLE has agents inject controlled mutations into working ML repositories, producing over 6,600 bug-injected variants and nearly 260K lines of injected code from 222 source repositories.
On BuggyMLE-Bench-Lite, a difficulty-graded repository-level version of MLE-Bench Lite, the top solve rate across 14 models fell from 95% on the easiest tier to 9% on the hardest.
A Qwen3.5-9B fine-tuned on a three-level curriculum from 200 more repositories rivaled frontier models on that benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →. Its SWE-benchThe standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.Full definition → Lite resolve rate rose from 24% to 33%, and LiveCodeBench v6 pass@1 from 69% to 71%.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.