Harness Design for Long-Running Application Development
- Source
- anthropic.com
- Author
- Anthropic Engineering
- Date
Context
A separate evaluator gives a coding agent concrete feedback to revise against, but it needs more than a request to judge quality. Anthropic's long-running development experiment paired a builder with an evaluator that interacted with the running application, tested behavior, and assessed explicit criteria. Before each chunk of work, the two agreed on what would count as completion.
That agreement connected a broad product specification to testable behavior. Failed criteria sent the work back with detailed feedback. For visual design, the researcher also the evaluator using examples, because a separate could still be overly generous. The useful mechanism is a defined review-and-revision loop whose standards can be inspected.
- calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
- LLM-as-judge — Using one model to score another's output against a rubric, so quality can be measured at a scale human grading cannot reach.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
“Harness design is key to performance at the frontier of agentic coding.”
“Taking inspiration from Generative Adversarial Networks (GANs), I designed a multi-agent structure with a generator and evaluator agent.”
“The final result was a three-agent architecture—planner, generator, and evaluator—that produced rich full-stack applications over multi-hour autonomous coding sessions.”
“Whether a layout feels polished or generic is a judgment call, and agents reliably skew positive when grading their own work.”
“Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue.”
Checking sign-in…
Loading comments…