How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner
- Source
- youtube.com
- Author
- AI Engineer
- Date
The VS Code team ships to a very large user base, which makes its switch from monthly to weekly releases a useful reference point. Kirschner describes the pieces that had to change together: a codebase prepared so agents can work in it, mandatory AI code review on changes, telemetry that opens auto-fix pull requests, staged rollouts and evals that feed lessons back into the process. He cites code survival, the share of written code that remains in place, rising from 55% to 86%. The talk is also candid that the move surfaced CI load and token-cost problems, so the speedup came with new operating costs. For teams considering a faster release cadence with agents, the takeaway is that review, rollout and measurement had to be rebuilt alongside code generation, not after it.
- Slow CI compounds when many agents hit it at once (the talk cites 20). VS Code's move to TypeScript Go made builds about 10x faster, part of making the codebase -ready alongside skills that encode team expertise.
- Agents get a self-correcting loop by using the app itself: VS Code lets them drive the product with Playwright and automated component screenshots so they can check their own changes.
- Quality is held with mandatory AI code review, AI issue triage and staged rollouts, and 51 billion daily telemetry events are turned into auto-filed issues and fix PRs.
- VS Code runs its own VSC-Bench . Kirschner reports that a 5-character 'hello world' eval cost one model 70x more .
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…