CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
- Source
- arxiv.org
- Author
- Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
- Date

Context
A static-analysis checker is a that scans source code for a particular defect pattern without running it. The CheckerBench paper, submitted October 6, 2026, asks whether an agent can write one from scratch: read a vulnerability description, inspect the repository, implement analyzer logic, and refine it using compiler and analysis feedback.
The has 300 tasks derived from 297 known vulnerabilities (CVEs) in 167 repositories. A companion framework rebuilds each submitted checker and measures whether it distinguishes the vulnerable version from the fixed one, along with patch localization and false positives. Across 21 and configurations with three repeats each, mean Pass@1, the share of tasks passed on a first attempt, was 32.30 percent and the best was 45.33 percent.
The authors conclude that reliable checker development remains challenging for current coding agents. Only the abstract was available, so the pass criteria are as summarized there.
- Checker synthesis asks an agent to interpret a defect specification, inspect the repository, write analyzer-specific logic and refine it through repeated compile-and-analyze feedback, which existing coding-agent benchmarks rarely assess end to end.
- Each task ships vulnerable and fixed revisions, a pinned analysis environment and a checker scaffold, spanning 85 CWEs.
- A companion framework, CheckerLab, independently rebuilds every submitted checker and also tracks how the agent used tools, alongside diagnostic contrast, localization and false positives.
- Across 21 model-harness configurations with three repeats each, mean Pass@1 was 32.30% and the best configuration reached 45.33%.
- Program — A set of instructions stored so that a computer can run it.
- Benchmark — A fixed set of tasks with a stated way of scoring them, used to compare models or setups.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- Harness — The software that sends a model its context, recognizes tool calls, runs or refuses them, returns results and keeps the record.
articleSWEeper-Bench: Can Agents Discover Bugs in Interactive Software?Yang Yao, Haozhe Chen, Bingyi Kang, Karthik R Narasimhan, Zhuang Liu
articleSWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding BenchmarkSoham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue Zhang
articleVerifierBench: Benchmarking the Generation of Executable Verifiers for LLM AgentsJianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang
Checking sign-in…
Loading comments…