RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving
- Source
- arxiv.org
- Author
- Xiyuan Zhou, Zhuoqi Li, Xinlei Wang, Yirui He, Yuhao Wu, Yuheng Cheng, Yan Xu, Junhua Zhao, Jinjin Gu
- Date
Context
RePro rewrites familiar mathematics problems while checking that their replacement answers have formal proofs. It generates candidate questions, translates them into a formal statement, searches for a proof and retains answers verified by the Lean proof checker. This addresses a problem with refreshed tests: a model’s apparent mistake may otherwise come from an invalid question or incorrect answer key.
The guarantee has a boundary. Lean checks the formal statement; matching that statement to the ordinary-language question still uses model-assisted screening. The reported accuracy changes reveal sensitivity to reformulation, while establishing memorization would require separate evidence about training-data overlap.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleCompiler-Guided Adaptive Proof Search with Cross-Model Synergy on Context-Dependent Theorem ProvingZhuo Liu, Ding Yu, Hangfeng He
articleVerified Learning for Compiler Optimization: An LLM-Guided Architecture with Formal ControlDev Pratap Singh, Rong Feng, Suman Saha
articleFinding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM AgentsYekun Chai, Qiwei Peng, Haoyi Xiong
Checking sign-in…
Loading comments…