Keep Evaluation Fair: Detecting Data Leakage in Code Generation Benchmarks via Membership Inference Attacks

A coding benchmark becomes less informative if its questions were already in a model's training data. CGMIA tries to identify that exposure by combining several clues, including similarity to a reference solution and whether generated code passes tests. It first trains a separate model on a known subset of benchmark examples, creating examples of exposed and unexposed data from which a classifier can learn. The researchers describe this as a membership-inference method.

They report stronger performance than existing methods in most comparisons across eight coding benchmarks. The output remains a prediction of training-set membership, rather than direct inspection of the target model's training records. Its useful role is to flag possible contamination when interpreting benchmark scores.

“Code generation benchmarks are widely used to evaluate Large Language Models (LLMs), but benchmark data leakage into training sets can inflate performance and undermine evaluation validity.”

“Experiments on eight code generation benchmarks show that CGMIA outperforms eight existing membership inference methods in most cases.”