Auto-Harness
github.com/neosigmaai/auto-harness- Category
- AI Agents
- Rank
- No. 1014Tools index
Previous survey · No. 1020 ·
- Pricing
- Open Source
- Type
- TOOL
- Builder
- neosigmaai
- GitHub
- 537 stars
- Date
About
A self-improving AI agent system that automatically mines failures from benchmarks, optimizes agent performance, and gates changes against regressions. It demonstrated improving agent scores from 0.56 to 0.78 on Tau3 benchmark tasks through autonomous iteration.
What it does
Auto-Harness gives a coding agent a controlled workspace for improving another agent. It runs benchmark tasks, preserves training traces, asks the coding agent to diagnose failures, and accepts edits only after regression and held-out checks. Successful iterations add newly solved tasks to the regression set and record results for later comparison.
Why it's ranked here
The design is unusually concrete: editable agent templates, benchmark adapters, trace capture, change isolation, regression checks, and held-out validation form one coherent loop. Support for customer service, terminal work, and interactive SQL shows useful range. Setup remains substantial, especially where external datasets, containers, credentials, or benchmark-specific services are required.
What's good
The gate checks both remembered failures and the full test split, so a narrow fix cannot pass merely by satisfying a handpicked case. Missing task results count as failures. Test traces are suppressed to reduce leakage. A file guard also confines iterative edits to the intended agent surface while leaving experiment records available for learning.
Tradeoffs
This is an experiment harness, not a ready autonomous agent product. Every supported benchmark brings its own infrastructure and credentials. Terminal tasks depend on Harbor and a container provider. Interactive SQL needs Docker, a separate dependency environment, an external repository, and privately requested gold data. Adding another benchmark still requires Python integration work and new templates.
How to use it well
Use it when you already have a scored benchmark, an editable Python agent, and enough compute to rerun training and held-out tasks repeatedly. Start from the supplied integration closest to your domain, inspect failure traces, and keep changes inside the guarded agent surface. It does not supply production deployment, end-user monitoring, or a substitute for representative evaluation data.
Technical notes+
prepare.py validates benchmark-specific credentials and tools, initializes experiment state, copies the selected template into agent/agent.py, composes PROGRAM.md, and runs a baseline. benchmark.py defines BenchmarkRunner plus TauBenchRunner, TerminalBenchRunner, and BirdInteractRunner; absent verifier output is represented as None and contributes zero to mean reward. gating.py uses ALLOWED_AGENT_FILES for a Git-backed edit guard, evaluates the maintained suite, always runs the full test split, and promotes newly passing training tasks only after both checks pass. record.py appends iteration metadata after checking the latest commit. agent/helpers/bird_interact/bird_service.py exposes FastAPI session endpoints backed by the ADK adapter in agent/helpers/bird_interact/bird_adk_runtime.py.
Observed
- Primary language
- Python
- Python requirement
- Python 3.12 or newer
- Packaging
- Setuptools build backend declared through pyproject metadata
- Install surface
- Python package dependencies include OpenAI, PyYAML, and tau2; Harbor is installed separately for terminal benchmarks
- Interfaces
- Python benchmark runner API, command-line preparation, gating and recording scripts, plus a FastAPI wrapper for BIRD-Interact
- Supported benchmark integrations
- tau-bench, Terminal-Bench 2.0, and BIRD-Interact
- Container support
- Docker is used for tau-bench and BIRD-Interact; Terminal-Bench supports Docker and external sandbox providers
Read from README.md, pyproject.toml, gating.py, record.py, prepare.py, benchmark.py, agent/agent.py, agent/helpers/__init__.py, agent/templates/tau_bench.py, agent/templates/bird_interact.py, agent/templates/terminal_bench.py, agent/helpers/bird_interact/setup.py, agent/helpers/bird_interact/__init__.py, agent/helpers/bird_interact/bird_service.py, agent/helpers/bird_interact/bird_adk_runtime.py.
What it can do
Mine failures from benchmark test results
Benchmark test results and failure logs → Categorized failure patterns and error analysis
Optimize coding agent performance through autonomous iteration
Coding agent and benchmark suite → Improved agent with higher benchmark scores
Automatically improve system prompts based on failure analysis
Agent failure data and current system prompts → Optimized system prompts
Gate code changes against regression testing
Proposed agent changes and evaluation suite → Approval or rejection of changes with regression analysis
Maintain and update evaluation test suites
Historical performance data and new test scenarios → Updated evaluation suite with relevant test cases
Run overnight autonomous agent improvement cycles
Initial coding agent configuration and target benchmarks → Iteratively improved agent versions with performance metrics
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.