HerAgent: A Block-Level Checkpointing Harness for Coding-Agent Environment Setup
- Source
- arxiv.org
- Author
- Xiang Li, Siyu Lu, Federica Sarro, Claire Le Goues, He Ye
- Date

Context
Before a coding can run a project's tests, something has to install the right system packages, language runtimes and dependencies, and HerAgent, from researchers at University College London, Uppsala University and Carnegie Mellon, makes that setup step more reliable by saving progress in validated stages. The difficulty is that a failure often appears long after the step that caused it. In the paper's example, an agent setting up the DataHub project installs Node 17 without complaint, but that version ships with OpenSSL 3, which breaks the frontend's build tooling, and the error only shows up several steps later when the frontend is checked. Existing setup agents either work one command at a time, so rolling back the last command leaves the wrong Node version in place, or regenerate a whole setup script after each failure, which throws away the parts that already worked.
HerAgent works at a middle size. After reading the repository, it plans setup as an ordered list of blocks, each a group of commands with one job, such as installing system packages, provisioning the language runtime, installing project dependencies or preparing the test tools, and each paired with a quick check like asking pytest to list the tests it can find. When a block runs and passes its check, HerAgent saves a snapshot of the container. When a block fails, it tries a fix inside that block, but it accepts the fix only after writing it back into the block's script and rerunning the block from the previous snapshot. If the evidence points to an earlier block, as in the Node case, it returns to the snapshot before that block and redoes the later ones. At the end it replays every block in a fresh container and counts the setup as successful only if that replay works, so a fix typed into the live container but never saved cannot produce a false success.
Each comparison used the same underlying model on both sides. Against the setup agent that three of the benchmarks were built for, and against the general-purpose OpenHands agent on SetupBench, HerAgent posted a higher success rate on all four: 52.0% to 85.0% against Installamatic, 46.3% to 70.4% against OpenHands, and smaller gains of 66.0% to 70.0% and 86.0% to 87.1% against ExecutionAgent and Repo2Run. On SetupBench it cost $0.12 per task against $2.20 for OpenHands. The comparisons are not perfectly like for like: HerAgent counts a task as solved if any of three runs succeeds, while the baseline figures are released results under each tool's own protocol, which the authors did not rerun, and Installamatic's protocol counts a task as solved if any of ten attempts succeeds. Measured one run at a time, HerAgent with GPT-4.1 solved between 48.1% and 64.8% of SetupBench tasks depending on the round. The paper's ablation on SetupBench, run with gpt-5.4 under identical conditions, is the more controlled evidence. Full HerAgent reached 90.7%; splitting the script into eight fixed chunks instead of functional blocks dropped it to 48.1%, never writing successful repairs back into the script dropped it to 25.9%, command-by-command execution with saved repairs reached 77.8%, and regenerating the whole script after failures reached 46.3%.
The authors judge success only by each 's own readiness check, which does not prove a repository is fully usable, and their benchmarks exclude projects that need private credentials, paid services or GPUs. Rolling back to an earlier block was rare but often decisive: it turned 6 of 7 such runs into successes on Repo2Run's benchmark and 5 of 9 on ExecutionAgent's. The failures that remain are mostly about dependencies. In the researchers' manual review of 904 failure records, missing dependencies and dependency conflicts together accounted for 63.1%, which the authors take as a case for setup agents that handle package compatibility explicitly. A separate October paper, SpecBuild, targets the same slow trial-and-error loop from another angle, running several candidate setup routes in parallel and keeping the first that works.
- Existing setup agents control progress per command or per whole script, the authors argue, so recovery is either too local to clear polluted state or too coarse to keep useful progress.
- HerAgent groups environment setup into semantic blocks and uses that one unit for execution, validation, checkpointing, repair and rollback.
- Against six prior agents on four environment-setup datasets, HerAgent improved setup success in every comparison, with the largest gains on Installamatic-Bench (52.0% to 85.0%) and SetupBench (46.3% to 70.4%), per the abstract.
- HerAgent also reduced setup costs by up to 94.5% in the authors' , alongside the success gains.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
articleAuditable Human-Agent Teams for Autonomous X-ray ExperimentationGunik Luthra, Ronald J. Pandolfi, Benjamin T. Diroll, Ricky Ronquillo, Ruipeng Li, Christopher B. Murray, Marcus M. Noack, Kevin G. Yager, Esther H. R. Tsai
articleEnvironment Steering: Using Data Flow Control to Improve Agent Utility and SafetyCharlie Summers, Prajwal Raghunath, Aaditya Pai, Mayur Kulkarni, Zhuo Zhang, Oliver Kennedy, Eugene Wu
articleLabAgent: Customize Any Research Hubs for Scientific Discoveries Using AI AgentsLei Liu, Yikun Zhang, Jialin Chen, Wanjia Zhao, Rex Ying, Wengong Jin, Hua Xu, James Zou, Tianyu Liu, Hongyu Zhao
Checking sign-in…
Loading comments…