Nvidia Nemotron 3 5 Super Vl Cline Agent Harness
- Source
- cline.bot
- Author
- Cline evaluation and engineering blog
- Date
Cline tested NVIDIA's Nemotron 3.5 Super VL, a 120B open-weight model with 12B active parameters, on Terminal-Bench 2.1 inside its own agent and three unnamed open-source harnesses. With thinking turned off, the model scored 61.8% in Cline against 53.9%, 51.7% and 44.9% elsewhere, and two of those harnesses ran with high reasoning or effort settings. Cline attributes the difference to three changes it made for open-weight models: a prompt that asks for a plan before code changes, a stall detector that ends a looping attempt and logs a structured event, and task checks that test the behavior the task depends on. Two run logs illustrate the gap. On a chess puzzle, Cline converted the board to FEN and ran an exhaustive search, while another harness reopened the board image repeatedly before timing out. Cline is clear about the limit: it did not isolate which change produced the gain, and it did not measure token savings. The result still matters for anyone comparing open models, because a 17-point spread on one model shows that a leaderboard score partly measures the harness around it.
- Cline notes it scored Nemotron 3.5 Super VL with thinking off on Terminal-Bench 2.1, while two of the three comparison harnesses used high reasoning or effort settings.
- Cline's stall detector ends an attempt that keeps repeating an action and records a structured event with the error class, whether it is recoverable, and the iteration number, which Cline says makes failed runs easier to debug than reading raw tool-call sequences.
- On the Terminal-Bench chess-best-move task, Cline converted the board to FEN and ran an exhaustive checkmate search that found two mates in one, while one competing made 29 tool calls, reopened the board image at least six times and timed out.
- On the headless-terminal task, Cline used an itemized checklist covering both whether Ctrl-C delivered a signal and whether the shell survived afterward, finishing in five commands; Cline says narrower checks in other harnesses missed the behavior that broke.
- Cline runs an internal Slack agent built on Nemotron that uses Cline to triage tasks and investigate user bug reports, can spin up sandboxes and containers, and escalates to more advanced models when a task needs them.
- agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…