Gives a repeatable loop (run standardized tasks, change one thing, keep or revert) for raising a coding agent's score, with the reported 47% to 57% gain as an example of what failure diagnosis can yield.
Key takeaways · AI-distilled
Cline ran Terminal Bench's 89 tasks through Harbor, which handles sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → setup, the agentic loopThe cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.Full definition →, verification and teardown; parallelizing on Modal brought a full run to about 35 to 45 minutes versus many hours sequentially in local Docker.
Failure analysis surfaced fixable Cline issues: the CLI's default 600s timeout was too short (raised to 2400s), the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → declared success without verifying files, command exit codes were not surfaced, and long-running commands were cut off.
The authors estimate at least 25% of failures cannot be fixed by config or prompt changes and need a better model; they call bucketing failures into model mistakes versus misconfiguration or missing logic the key agent skillA reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.Full definition →.
Cline reports scores vary several points run to run: six runs of one Opus 4.5 config scored 0.43 to 0.49 (average 0.458), so they suggest 3 to 6 repeat runs before trusting a close comparison.
To A/B test agent code without merging, Cline's Harbor adapter accepts github_user and commit_hash agent kwargs, letting the same benchmark run against main and a PR branch for direct score comparison.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
agentic loop — The cycle an agent runs in: decide, call a tool, read the result, decide again — repeating until the goal is met or a stop condition fires.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.
