ALE-Bench
sakana.ai- Category
- Developer Tools
- Rank
- No. 2192Tools index
Previous survey · No. 2064 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- Date
About
Sakana AI's benchmark of long-horizon algorithm engineering on hard combinatorial optimization problems from programming contests. Scores are contest-style ratings, so compare the time budget and iteration setup.
What it does
Sakana AI's page describes ALE-Bench as a coding benchmark built with AtCoder from 40 past AtCoder Heuristic Contest problems, hard optimization tasks with no known optimum. It says the benchmark supplies problem statements, visualizers, a code execution environment and ranking software, so an AI system can iterate on solutions as a human contestant would.
Stated on the product site
- Task set
- 40 optimization problems from past AtCoder Heuristic Contests
- Provided tooling
- Problem statements, visualization tools, a code execution environment and ranking software
- Scope of systems evaluated
- From bare LLMs to scaffolded agents
- Partner
- Developed in partnership with AtCoder Inc.
- Published artifacts
- Technical report on arXiv, dataset on Hugging Face, code on GitHub
- Reported evaluation setting
- Models refined solutions with self-refinement within four hours
Not stated on the site
- Which license covers the dataset and code is not stated on the page.
- How the contest-style ranking is computed for an AI submission, beyond the reference to ranking software, is left to the paper.
- Whether results under different time budgets or iteration counts can be compared directly is not explained on the page.
- Whether the 40-problem set is updated as new contests run is not stated.
Written from the product site at sakana.ai.
Tags
benchmarkevaluation
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.