I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft
Source
youtube.com
Author
AI Engineer
Date
Why it matters
Shows concrete failure modes of unsupervised coding-agent refactors (tests hitting production inference, labels that don't match behavior) and a workflow of owning goals and evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → rubrics while delegating implementation.
Key takeaways · AI-distilled
A bad digest sent Keiji Kanazawa looking for a trace, and he found his supposed AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → was still a deterministic workflow: skills had no agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition → and a daily reflection job was producing empty reports.
Kanazawa says he had approved agent plans without checking whether the implementation meant what its labels promised, and one agent ran unit tests against production inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → about 30 times.
His rebuilt example is a video deep-dive agent that extracts 440 keyframes from an inference lecture, pairs blackboard equations with their explanations and lets him question individual sections.
He compared four agent frameworks before choosing tools for his own needs as a product manager, and says the keyframe approach generalizes to slide decks, coding demos and terminal recordings.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
videoYour KV Cache Can Spot Attacks on Your AI Agents - Ari Azarafrooz, Intrinsec AI
videoWill AI Agents Pretend to Be Human to Get Past a Check? - Maty Bohacek, Stanford
videoThe One Thing Every AI Agent Framework Agrees On - Mislav Ivanda, Daytona
videoAI Writes Code Faster Than Uber Can Merge It - Preetam Dwivedi & Manjari Akella, Uber
