Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling
- Source
- youtube.com
- Author
- Stanford Online
- Date
Context
Repeated sampling means asking a model the same question many times and keeping an answer that passes a check. A Stanford CS329A lecture covers the Large Language Monkeys result: coverage, the share of problems solved by at least one attempt, keeps rising as samples grow from one to a thousand. In the paper, DeepSeek-Coder-V2-Instruct's solve rate on Lite rises from 15.9 percent with one sample to 56 percent with 250, above the 43 percent single-attempt state of the art cited in the abstract. The lecture presents this as a way for open models such as DeepSeek's to beat larger ones.
The lecturer's main caveat is verification. Extra coverage becomes a better answer only when something can identify the correct sample, such as unit tests for code or a proof checker. The paper's abstract says that where no automatic verifier exists, majority voting and reward models plateau beyond several hundred samples. The course's closing lecture, summarized in Intel's Part 9 entry, returns to test-time scaling among other directions.
- SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
- test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
videoStanford Webinar - Building Trustworthy AI: Navigating Security in the Agentic Era
videoStanford CS547 HCI Seminar | Autumn 2025 | HCI Systems in the Age of AI Code Generation
videoStanford CS25: Transformers United V6 I Advancing Science and Medicine with Collaborative AI Agents
videoStanford CS153 Frontier Systems | Andreas Blattmann from Black Forest Labs on Visual Intelligence
Checking sign-in…
Loading comments…


