How We Migrated 11 Million Users To Clines Biggest Harness Upgrade
Source
cline.bot
Author
Cline evaluation and engineering blog
Date
Why it matters
Shows how to roll out a large AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →-harness change to millions of users without staged marketplace releases, including what broke the first time. Useful for anyone shipping agent updates at scale.
Key takeaways · AI-distilled
Cline shipped the legacy and new extensions in one package behind a small loader that picks a bundle per window from a cached PostHog flag, falls back to legacy and pins the machine if the new one fails activation, and uses the flag as a kill switch.
Since one VS Code extension gets one manifest, Cline's build generates a union package.json from both bundles, gates single-bundle commands behind a loader-set context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → key, and hard-fails if views or settings schemas diverge.
Cline recommends instrumenting the old system before comparing: they added task.mistake_limit_reached to the legacy agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → with identical semantics, then held near a 50/50 split for over a week as their cleanest A/B window.
With cohorts near 50/50, tasks hitting the three-mistakes-in-a-row limit fell from 6.34% on the old harness to 0.62% on the new one, with per-model drops ranging from 6x to 11x.
Cline attributes the gap to the 2024 harness parsing XML tool calls from text, built for Claude 3.5 Sonnet, which taxes 2026 models trained to call tools natively with format errors, parse failures, and retries.
Terms in this piece · Glossary
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
