← All IntelClip / AI AgentsWhy current coding benchmarks miss the real job
From Emulated: The Data for Fully Autonomous Software Engineers and Companies — Joseph Wang · ≈2:55
“the tasks only operate within the code base.”
“We've taken software engineering companies and we've put them into containerized environments.”
“environments are far more complex and long horizon than a simple code diff.”
What’s in it
- Explains why current coding benchmarks miss real engineering work
- Introduces containerized environments simulating full software companies for agent training
- Covers agent challenges like distributed systems failures and operational blast radius
Clip transcript
right now? And how is this influencing the model capability gap here? So, if you look at any of the frontier or recent benchmarks, like SweBench Pro, Terminal Bench, or something like Frontier Code and Deep Sweep, um the tasks only operate within the code base. Uh the agent is given a pretty large uh task uh and over the course of 50 to 100 turns produces a couple thousand-line PR. Um but it doesn't do all of the work that a human does. It doesn't do uh what a PM does with talking to customers, understanding their problems, what an engineer does with trying out different approaches, performing performance testing them, um and owning the underlying infra for the code base over the course of not just months, but years. >> And this is really the gap that we're closing. We've taken software engineering companies and we've put them into containerized environments. so this includes uh include like organizational contexts like projects, incidents, customer conversations. Uh the agent also has to deal with issues that only appear at scale like network failures between distributed nodes, data corruption, and clock skew. And through all this, we also want the agents to reason about orchestrating through distributed clusters and also thinking about things like operational blast radius while solving live traffic. And the result is that the task that these agents have to complete or we want the agents to learn is that environments are far more complex and long horizon than a simple code diff.
Comments
Sign in to comment.
Loading comments…