Vibeleaderboard
← All Intel
Intel / video

Evaluating and improving Replit Agent at scale

Source
youtube.com
Author
Claude
Date

Context

Replit, whose users describe an app in plain English and expect it to work, says standard coding benchmarks do not fit its situation. In a talk from Replit's president and head of AI, the argument is that tests like apply a patch to an existing repository and run existing tests, while Replit users start from an empty codebase and write no tests. Its new open benchmark gives an agent one of 20 product requirements documents, plain-English app descriptions drawn from real usage, and has an evaluator agent open the built app in a browser and walk through a natural-language test plan, scoring on steps that fail.

Variants cover building from scratch, adding a feature to a working reference app, extending an agent-built app, and parallel agent work. The speaker reports a nearly 2x gap between frontier and models, without saying which measure it applies to, and that most models do worse when extending their own code. These are Replit's findings from its own benchmark.

Terms in this piece · Glossary
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
  • Open weights — A model whose learned parameter values have been made available to download, under terms the publisher sets.
  • Prompt — Information or instructions supplied to a model; in a chat, the request you type is often called your prompt.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from Claude
Recommended reads
Comments

Checking sign-in…

Loading comments…