Intel / article

Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite

Source: arxiv.org
Author: Mohsen Arjmandi
Date: Published Sep 14, 2026

Context

Using a model vendor's own software did not produce a clear average advantage in this coding study. A single-author study held the model fixed while comparing its vendor's , the prompts, tools and control flow around it, with a multiprovider alternative. It tested Anthropic's Opus 4.8 and OpenAI's GPT-5.5 on the same selected 80 tasks. The paper reports average differences of just 1.25 percentage points in opposite directions.

The uncertainty intervals allow meaningful advantages for either harness. The 80 tasks were also deliberately selected from a larger private suite, with tasks on which two screening runs disagreed over-represented, so the averages describe that pool rather than all coding work. The result challenges an automatic assumption that vendor-native software performs better, while leaving the choice dependent on the workload and configuration.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Read the source arxiv.org