Contamination in SWE-bench Verified via Opus memorization
www.youtube.com- Category
- Education
- Type
- SEGMENT
- Added
- Aug 11, 2026
About
Concrete evidence that a top model has memorized a major coding benchmark's contents, undermining reported scores and highlighting the industry's lack of contamination disclosure.
Why it made the leaderboard
Concrete evidence that a top model has memorized a major coding benchmark's contents, undermining reported scores and highlighting the industry's lack of contamination disclosure.
Comments (0)
No comments yet
Indexed by a proprietary survey. Corrections welcome.