Shows how to let an LLM propose program invariants safely, with guards against side effects and infinite loops in generated checks, useful for anyone running LLM-written code at runtime.
Key takeaways · AI-distilled
Oca lets an LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → propose candidate invariants instead of drawing only from a fixed template library like Daikon's. On average 28% of its invariants that held at run time call a project-specific method, a form Daikon's templates cannot express.
In manual inspection on open-source Java projects, 18% of Oca's invariants were false positives versus 76% for Daikon. Used as properties for property-based testing, Oca's invariants exposed 9 of 25 real bugs to Daikon's 4.
Oca was also cheaper in the authors' comparison: 566 minutes and 27 GB of disk, against 2,817 minutes and 827 GB for Daikon.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleSoundness Checking of Taint Flow ModelsSamarth Kishor, Victor Nicolet, Joey Dodds
articleVerified Learning for Compiler Optimization: An LLM-Guided Architecture with Formal ControlDev Pratap Singh, Rong Feng, Suman Saha
articleWhen Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic SystemsShihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie