ICALL-BENCH: Benchmarking LLM Interprocedural Reasoning in Binary Programs
Source
arxiv.org
Author
Shenglong Yao, Zirui Liu, Xiangchi Yuan, Wenke Lee
Date
Why it matters
Existing binary benchmarks can be solved from local clues. This one measures cross-function reasoning on large real programs, showing where LLMs and agents fall short in reverse engineering.
Key takeaways · AI-distilled
ICALL-BENCH uses indirect-call target recovery as its probe for interprocedural reasoning, across 50 complete real-world C/C++ programs averaging 74,746 lines of code and 941 functions, built under multiple optimization configurations.
The benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → separates three regimes: model-only reasoning, agents with static analysis tools, and Internet-enabled agents, so intrinsic reasoning can be told apart from information gathering and outside knowledge.
Across five recent LLMs and their agents, program-level verifiable success ranged from 23% to 81%. More information access did not consistently help, with the smallest gains on the hardest cases, the authors report.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleNavigating Epistemic Parity in LLM Agents: A Benchmark for Cross-Source Conflict Resolution Between Memory and Procedural SkillsPrakhar Pandey
articlePerfReasoning: How Well Do LLMs Reason on Hardware Performance?Dan Zhao, Karthikeyan Sankaralingam, Christos Kozyrakis, Qijing Huang
articleVerifierBench: Benchmarking the Generation of Executable Verifiers for LLM AgentsJianing Yin, Ziyi Chen, Hao Wang, Yiyou Sun, Zijian Jin, Dawn Song, Daoan Zhang