Vibeleaderboard
← All Intel
Intel / article

CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

Source
arxiv.org
Author
Hang He, Li Wang, Hao Chen, Yuchen Shao, Yuling Shi, Lisheng Wang, Peiyang Liu, Goose Lin, Zaiyuan Wang, Haiying Sun, Ting Su, Chengcheng Wan
Date

Context

A static-analysis checker is a that scans source code for a particular defect pattern without running it. The CheckerBench paper, submitted October 6, 2026, asks whether an agent can write one from scratch: read a vulnerability description, inspect the repository, implement analyzer logic, and refine it using compiler and analysis feedback.

The has 300 tasks derived from 297 known vulnerabilities (CVEs) in 167 repositories. A companion framework rebuilds each submitted checker and measures whether it distinguishes the vulnerable version from the fixed one, along with patch localization and false positives. Across 21 and configurations with three repeats each, mean Pass@1, the share of tasks passed on a first attempt, was 32.30 percent and the best was 45.33 percent.

The authors conclude that reliable checker development remains challenging for current coding agents. Only the abstract was available, so the pass criteria are as summarized there.

Key takeaways · AI-distilled
  • Checker synthesis asks an agent to interpret a defect specification, inspect the repository, write analyzer-specific logic and refine it through repeated compile-and-analyze feedback, which existing coding-agent benchmarks rarely assess end to end.
  • Each task ships vulnerable and fixed revisions, a pinned analysis environment and a checker scaffold, spanning 85 CWEs.
  • A companion framework, CheckerLab, independently rebuilds every submitted checker and also tracks how the agent used tools, alongside diagnostic contrast, localization and false positives.
  • Across 21 model-harness configurations with three repeats each, mean Pass@1 was 32.30% and the best configuration reached 45.33%.
Terms in this piece · Glossary
  • Program — A set of instructions stored so that a computer can run it.
  • Benchmark — A fixed set of tasks with a stated way of scoring them, used to compare models or setups.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • Harness — The software that sends a model its context, recognizes tool calls, runs or refuses them, returns results and keeps the record.
Recommended reads
Comments

Checking sign-in…

Loading comments…