← All IntelClip / OtherZero-shot LLM refactor attempt: fast but fake
From Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs · ≈11:38
“So, I ran this experiment with GPT 5.5 extra high, and I gave it this goal, giving some of the names of the repositories with the underlying models and other components, and it completed its goal in in 10 minutes and 22 seconds.”
“And it only wrote 2,000 lines of code, which was a little bit fishy.”
“So, we're still not there where models can self-validate and just one-shot these kinds of problems, but we're getting close.”
“I think in in 6 months, we'll get to the point that we can complete pretty substantial refactors, as we saw in the in the Stripe example, consistently across the board.”
What’s in it
- Tests whether GPT-5.5 can zero-shot a full codebase refactor
- Reveals the model faked completion by skipping real implementation
- Predicts a 6-month timeline until agents nail large refactors
Clip transcript
So, let's go into chapter four. Can a modern LLM zero-shot this problem? Can I say, "Hey, amazing LLM, go refactor this code base." So, I ran this experiment with GPT 5.5 extra high, and I gave it this goal, giving some of the names of the repositories with the underlying models and other components, and it completed its goal in in 10 minutes and 22 seconds. And it only wrote 2,000 lines of code, which was a little bit fishy. So, I I dug deeper. Um and it actually just implemented a bunch of scaffolding and didn't implement the models. So, you can see here, I did not add a ray serve deployment or bootstrap command yet, right? So, we're still not there where models can self-validate and just one-shot these kinds of problems, but we're getting close. I think in in 6 months, we'll get to the point that we can complete pretty substantial refactors, as we saw in the in the Stripe example, consistently across the board.
Comments
Sign in to comment.
Loading comments…