[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Clarifies what SWE-bench variants measure and why single-file benchmarks like HumanEval stopped discriminating. Engineers comparing coding agents can read published scores more critically.

• SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
• multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

• videoWorld Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
• videoExtreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI
• videoInside Abridge: The AI Listening to 100 Million Doctor Visits — Abridge's Janie Lee & Chai Asawa
• videoExo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research

• videoThe End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier EvalsLatent Space
• postFrom SWE-bench to ProgramBench: The Next Generation of Coding EvalsStanford AI Lab
• videoFullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, ConvexLatent Space

Checking sign-in…

Loading comments…