Why code retrieval benchmarks mislead, and how to rebuild them
- Source
- x.com
- Date
Vector-based code retrieval is a critical building block in code assistants and agents. However, many people complained about the lack of diverse, high-quality evaluation datasets for it. We surveyed existing ones and proposed some methods to build better ones. 🧵🧵
We identify 3 main issues with existing datasets: noisy labels, simplism, and data contamination risks (see above). We propose two methods to create new datasets, repurposing QA datasets and leveraging the issue/ticket record, and use them to build many new datasets below.
We evaluated various embedding models, @OpenAI , @awscloud CodeSage, CodeRankEmbed, @JinaAI_ v2 code, along with the @Voyage AI’s newly released voyage-code-3 (https://t.co/0lL99O7eSR) on these datasets:
Please check out a longer version in our blog post on code evaluation: https://t.co/WylTF7m5Kz
Context
A code-search test can reward matching copied wording without testing whether a system finds the code needed to solve a problem. Voyage AI's evaluation analysis identifies this weakness in benchmarks whose queries repeat descriptions already attached to the code.
One alternative it describes uses an issue report as the search query and the files changed to resolve that issue as the expected results. Another turns programming questions into queries and their solutions into the searchable collection. Both connect the retrieval task to a concrete problem and a known response, giving evaluators a way to test more than verbal similarity.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…