HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills
Source
arxiv.org
Author
Yakup Koray Budanaz, Lorenzo Paleari, Marcin Copik, Alexandru Calotoiu, Timo Schneider, Benjamin Chen Ming Choong, Shriram Chandran, Lars Hulsbergen, Levi Widmer, Torsten Hoefler
Date
Why it matters
Lets you test whether adding a tool or agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → to a code-optimizing AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → pays for its cost, with an external judge that blocks the shortcuts agents use to game timing and correctness checks.
Key takeaways · AI-distilled
Each kernel is specified once in NumPy, with C and Fortran references derived from it. Submissions come through a common C ABI or Python interface and run on CPUs or GPUs, single or multi-node.
The 689 kernels span three tracks: loop-level optimization challenges, kernels extracted from HPC codes, and machine-learning models.
The authors report agents reaching up to 4.2x speedup over automatic parallelizers, and 93 to 99% of answers without device-specific code stayed correct on another machine, with speedup rank correlation of at least 0.84.
Across 40 matched setups, adding interventions such as tools significantly changed speedup only twice but changed tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → cost by up to 3.0x.
Because the weights given to input, cached-input and output tokens decide whether a saving counts as significant, the authors state their weights and release every token count.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
articleKernelArc: A Multi-Agent Framework for GPU Kernel OptimizationJoyjit Kundu, Ben Stoffelen, Kaili Wang, Peter Vrancx, Ludovic Denoyer
articleHow Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull RequestsHuiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
articleCode Understanding is a Bottleneck for Coding AgentsNishant Balepur, Kiran Tomlinson, Tobias Schnabel