Vibeleaderboard
← All Intel
Intel / video

Stanford CS329A Self-Improving AI Agents | Part 2 | Test-Time Compute Scaling

Source
youtube.com
Author
Stanford Online
Date

Context

Repeated sampling means asking a model the same question many times and keeping an answer that passes a check. A Stanford CS329A lecture covers the Large Language Monkeys result: coverage, the share of problems solved by at least one attempt, keeps rising as samples grow from one to a thousand. In the paper, DeepSeek-Coder-V2-Instruct's solve rate on Lite rises from 15.9 percent with one sample to 56 percent with 250, above the 43 percent single-attempt state of the art cited in the abstract. The lecture presents this as a way for open models such as DeepSeek's to beat larger ones.

The lecturer's main caveat is verification. Extra coverage becomes a better answer only when something can identify the correct sample, such as unit tests for code or a proof checker. The paper's abstract says that where no automatic verifier exists, majority voting and reward models plateau beyond several hundred samples. The course's closing lecture, summarized in Intel's Part 9 entry, returns to test-time scaling among other directions.

Terms in this piece · Glossary
  • SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
  • test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
Read the source www.youtube.com
More from Stanford Online
Recommended reads
Comments

Checking sign-in…

Loading comments…