Small requirement changes to familiar DeepSWE tasks cause large score drops, because models guess what hidden tests expect. A high benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → score may not mean an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → follows your actual spec.
Terms in this piece · Glossary
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
