ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Source
Aaryan Kapoor, Md Abdullah Al Hafiz Khan
Author
Aaryan Kapoor, Md Abdullah Al Hafiz Khan
Date
Terms in this piece · Glossary
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
Why it matters
Code embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → are usually scored on topical similarity, which hides whether they can separate correct code from a near-identical buggy variant. This benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → plants execution-verified buggy distractors in the search pool to test exactly that.
