AutoAgent
github.com/kevinrgu/autoagent- Category
- AI Agents
- Rank
- No. 439Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- kevinrgu
- GitHub
- 4.6k stars
- Date
About
An AI meta-agent that autonomously builds and optimizes AI agent harnesses overnight. It iteratively modifies system prompts, tools, and configurations, runs benchmarks, and hill-climbs on performance scores without manual intervention.
What it does
AutoAgent turns agent development into a controlled experiment loop. You write a directive and supply Harbor-format tasks with verifiers. A coding agent then changes the tested harness, runs those tasks, studies failures, logs outcomes, and keeps only changes that improve passed tasks or preserve results with simpler code.
Why it's ranked here
Worth considering as a compact, inspectable agent-engineering laboratory. The editable harness is deliberately small, evaluation uses task-specific verifiers, and trajectories expose model activity and token usage. Its usefulness depends heavily on work supplied by the user: the repository includes no benchmark tasks, and reliable experiments require Docker, credentials, and carefully designed tests.
What's good
The design separates the editable agent from a fixed benchmark adapter, reducing accidental changes to evaluation plumbing. Docker contains task execution. Harbor compatibility makes task suites portable across harness experiments. The instructions also favor broad capability improvements over benchmark-specific tricks, require an untouched baseline, preserve discarded-run evidence, and treat equal performance with simpler code as progress.
Tradeoffs
This is infrastructure for experiments, not a ready-made benchmark suite. Users must create or import tasks, verifiers, container environments, and reference files. Repeated runs consume model resources and accumulate Docker images and containers. The stated Python requirement is inconsistent: the quick start says 3.10 or newer, while package metadata requires 3.12 or newer. Results are only as trustworthy as the supplied verifiers.
How to use it well
Use it when an agent engineer has representative, automatically verifiable tasks and wants disciplined comparisons of prompts, tools, routing, or sub-agents. Start with the required baseline, change one general capability at a time, inspect trajectories, and retain a clear experiment ledger. It does not replace benchmark design, production deployment, provider credentials, or routine Docker maintenance.
Technical notes+
The main editable surface in agent.py contains SYSTEM_PROMPT, MODEL, MAX_TURNS, create_tools, create_agent, and run_task; the fixed section implements the Harbor AutoAgent adapter and serializes ATIF-v1.6 trajectories. program.md defines baseline-first experimentation, pass-count comparison, simplicity tie-breaking, failure grouping, and keep-or-discard rules. pyproject.toml requires Python 3.12 or newer and declares openai-agents, pandas, openpyxl, numpy, and harbor. .python-version selects Python 3.13, while Dockerfile.base uses a Python 3.12 uv image and copies only agent.py. agent-claude.py contains an alternative Claude SDK harness, but its SDK and dotenv imports are not declared in pyproject.toml, and Dockerfile.base does not copy that file.
Observed
- License
- MIT
- Primary language
- Python
- Packaging
- Python project metadata with dependencies installed through uv
- Runtime requirement
- Package metadata requires Python 3.12 or newer; Docker is required by the documented workflow
- Interface
- Harbor-compatible agent adapter with ATIF trajectory output
- Benchmark surface
- Uses Harbor-format task directories with executable verifiers
- Repository payload
- The repository ships without benchmark tasks
- Harness structure
- Single-file primary harness split into editable logic and a fixed adapter boundary
Read from README.md, pyproject.toml, agent.py, agent-claude.py, program.md, .python-version, Dockerfile.base.
What it can do
Build AI agent harnesses autonomously
Task description → Optimized AI agent configuration
Modify system prompts iteratively
Initial prompts and performance feedback → Optimized system prompts
Configure agent tools automatically
Available tools and task requirements → Optimized tool configurations
Run performance benchmarks
Agent configurations → Performance scores and metrics
Optimize agent performance using hill-climbing
Performance scores and current configurations → Improved agent configurations
Iterate on agent development overnight
Initial agent setup and target metrics → Fully optimized agent system
Intel on AutoAgent
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.