Most coding benchmarks are puzzle-style. This sources reproducible environments from real repository failures, so you can see where agents actually break on engineering work.
Key takeaways · AI-distilled
cline-bench tasks come from opted-in Cline Provider usage on open source projects where the model needed manual intervention or could not finish, plus manual contributions. Private repositories are ineligible, and Teams and Enterprise usage is excluded.
Each published task includes the starting git commit of an open source repo, the original prompt (possibly lightly edited for fairness or sensitive data), and tests derived from the code the developer actually committed.
Tasks are packaged as reproducible environments following open specifications such as Harbor (Terminal-Bench 2.0) and Prime Intellect's Environments Hub, so they can serve as evaluations and as RL or fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → environments.
Cline announced a $1M sponsorship program paying selected task contributors in Open Source Builder Credits, and says contributors whose tasks are selected are attributed unless they ask to be removed.
Terms in this piece · Glossary
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
articleREAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production UsageSmriti Jha, Matteo Paltenghi, Chandra Maddila, Vijayaraghavan Murali, Shubham Ugare, Satish Chandra
articleLoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding AgentHan Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang