Vibeleaderboard
← All Intel
Intel / post

LLMs still produce bugs, but those bugs are different than what they used to be.

Source
x.com
Date
Boris Cherny@bcherny

LLMs still produce bugs, but those bugs are different than what they used to be. It’s less off-by-ones and more about system design, ui usability, missing broader context. Some kinds of coding has been solved, but not all. While models continue to improve, adversarial code review has been an incredibly powerful tool to catch many of these kinds of bugs. It can be as simple as a one line prompt - “use a dynamic workflow to adversarial test every edge case in an iOS simulator”, or use Claude’s built in /code-review (or /code-review low, /code-review medium, etc.)

Context

Boris Cherny argues that reviewing AI-written code increasingly requires to system design, usability and missing . In his account, these problems have become more prominent relative to small implementation errors.

He recommends adversarial review: deliberately trying to expose weaknesses and edge cases instead of accepting the first working implementation. That recommendation follows from his diagnosis, because locally correct code can still fit the wider application poorly. The post offers an experience-based review strategy, without measuring a general change in bug rates.

Terms in this piece · Glossary
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
More from Boris Cherny
Recommended reads
Comments

Checking sign-in…

Loading comments…