Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints
Source
arxiv.org
Author
Saransh Dhage
Date
Context
Controls around an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can still leave its planning step unpredictable. In a study using two models and two synthetic tasks, an initial execution agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → made repeated runs less reproducible in two of four combinations. The researcher then required plans to pass a fixed structural check before tools could run. Three combinations reached perfect reported determinismWhether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.Full definition → across 100 runs. The useful finding is that constraining tool execution alone can leave an uncontrolled planning stage as a source of variation. These were synthetic finance and legal tasks, and the speed effects differed by model: one became faster while the other became slower. Study abstract.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
determinism — Whether the same input reliably produces the same output — something LLM systems mostly lack, which changes how you test and debug them.
articleBeyond the Model: Demystifying Harness Effects in Software Engineering AgentsHaichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang Xiao
articleHarness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven SystemsPaul Barbaste, Tristan Darrigol, Germain Vu, Tom Wiltberger
articleHarness Engineering in LLM Tool Use via Agent-Native Reusable Tool PrimitivesHaibo Jin, Suijin Wang, Xucheng Yu, Haojing Luo, Haohan Wang