Vibeleaderboard
← All Intel
Intel / article

BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

Source
arxiv.org
Author
John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy, Max Xu, Sam Schmidgall, Mark Malhotra, Shwetak Patel, Jimeng Sun, Daniel McDuff, Bo Dai, Xin Liu
Date
Why it matters

Coding benchmarks are saturating; this gives a structural difficulty predictor and a way to generate harder repo-level debugging tasks for evaluating ML engineering agents.

Key takeaways · AI-distilled
  • The Code Complexity Index scores tasks on ten structural features, including repository scale, dependency depth and patch size. It predicted average solve rates across eight public coding benchmarks with R² = 0.951, with patch size and complexity as the top predictors.
  • BuggyMLE has agents inject controlled mutations into working ML repositories, producing over 6,600 bug-injected variants and nearly 260K lines of injected code from 222 source repositories.
  • On BuggyMLE-Bench-Lite, a difficulty-graded repository-level version of MLE-Bench Lite, the top solve rate across 14 models fell from 95% on the easiest tier to 9% on the hardest.
  • A Qwen3.5-9B fine-tuned on a three-level curriculum from 200 more repositories rivaled frontier models on that . Its Lite resolve rate rose from 24% to 33%, and LiveCodeBench v6 pass@1 from 69% to 71%.
Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
Recommended reads
Comments

Checking sign-in…

Loading comments…