autoresearch
https://github.com/karpathy/autoresearch- Category
- AI Agents
- Rank
- No. 185Tools index
- Listed in
- #6 Research with an agent
- Pricing
- Open Source
- Type
- AGENT
- Builder
- karpathy
- GitHub
- 95.4k stars
- Date
About
Andrej Karpathy's experiment: AI agents that autonomously run research on single-GPU nanochat training, proposing hyperparameter tweaks and summarizing findings.
What it does
This is a compact test bench where a coding agent changes one training program, runs each candidate under an equal wall-clock budget, and keeps only improvements. Humans shape the agent’s instructions, while a fixed preparation and evaluation layer supplies data, tokenization, and validation bits per byte.
Why it's ranked here
The design is unusually legible: one editable training surface, one fixed metric, and an explicit keep-or-revert loop. That makes experiments easy to compare on the same machine and diffs easy to inspect. Its narrow hardware target and intentionally minimal baseline make it more useful as a research scaffold than a general training system.
What's good
A fixed five-minute training window controls experimental cost even when architecture, model size, optimizer, or batch size changes. Validation bits per byte remains independent of vocabulary size. Failed and rejected ideas are logged alongside retained changes, including peak memory, while the single editable training surface keeps the agent’s work reviewable.
Tradeoffs
The official code requires one NVIDIA GPU and was tested on an H100. Results from different compute platforms are not directly comparable because the fixed wall-clock budget rewards hardware-specific throughput. There is no distributed training or broad device abstraction. The agent instructions also require unrestricted experimentation to stay inside existing dependencies and a fixed evaluation harness.
How to use it well
Use it when you have an NVIDIA GPU, understand training code, and want a controlled overnight search over model and optimizer ideas. Establish the unchanged baseline first, inspect retained diffs, and treat the experiment log as the audit trail. It does not replace distributed training infrastructure, cross-hardware benchmarking, or support for CPU, Apple silicon, Windows, and AMD hardware.
Technical notes+
program.md defines the autonomous branch, commit, run, log, keep, discard, and revert protocol, with results.tsv left untracked. train.py is the sole experimental target and contains the GPT model, Flash Attention 3 selection, Muon plus AdamW optimization, and training loop. prepare.py fixes TIME_BUDGET at 300 seconds, MAX_SEQ_LEN at 2048, the tokenizer, data loading, and evaluate_bpb; it downloads Parquet shards from Hugging Face and caches artifacts under ~/.cache/autoresearch/. pyproject.toml requires Python 3.10 or newer, pins torch==2.9.1, and configures the CUDA 12.8 PyTorch index. analysis.ipynb reads experiment TSV output and plots kept, discarded, and crashed runs.
Observed
- License
- MIT
- Primary language
- Python
- Installation surface
- uv project using pyproject.toml dependencies
- Interface
- Script-based workflow driven by a coding agent; no library or network API is described
- Platform support
- Official code requires a single NVIDIA GPU
- Python support
- Python 3.10 or newer
- Training dependency
- PyTorch 2.9.1 from the CUDA 12.8 package index
- Repository structure
- One agent-editable training script, one fixed preparation script, agent instructions, and an analysis notebook
Read from README.md, pyproject.toml, train.py, prepare.py, program.md, analysis.ipynb, .python-version.
What it can do
Run autonomous research experiments on nanochat training
Single-GPU environment and research parameters → Research experiment results and data
Propose hyperparameter optimizations
Current model training configuration and performance metrics → Recommended hyperparameter adjustments
Summarize research findings
Experimental data and training results → Research summary reports
Train nanochat models on single GPU
Training data and model configuration → Trained nanochat model
Analyze training performance metrics
Model training logs and performance data → Performance analysis and insights
Intel on autoresearch
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.