Intel / video

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

Source: youtube.com
Author: AI Engineer
Date: Published Jul 26, 2026

Context

DeepSWE tries to make coding tests harder to solve by recovering an existing answer. Its creators' presentation describes 113 newly authored tasks across 91 repositories, rather than tasks reconstructed from already completed public changes.

The also separates the 's environment from the software that grades its work. Version 1.1 removes repository history beyond the starting commit, reducing opportunities to retrieve a solution from that history. These measures address distinct routes to answer leakage. The resulting scores still describe performance under the benchmark's chosen agent setup and task mix, which the speaker acknowledges underrepresents bug localization and refactoring.

Terms in this piece · Glossary

benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Key quotes

“they're able to directly run git log and then go through the commit hashes and cherrypick the ones out that contain the golden patches”
James Shi

“it will go ahead and implement the synchronous part, but it may drop the asynchronous part. We observed this in roughly two out of three cloud rollouts”
James Shi

“you're not going to be coming in there with a to-do list uh telling it to oh first do this and then do this”
James Shi

“we find that the average size of our solution is five times the lines of code”
James Shi

“stronger models have a great tendency to want to test their own work”
James Shi

Read the source www.youtube.com