What Happened When I Applied Karpathy's Autoresearch Idea to LLM Inference
Source
Manthan Gupta
Author
Manthan Gupta
Date
Key takeaways · AI-distilled
The agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → splits into two files: prepare.py fixes the model, prompts, warmups and quality gates and is read-only, while inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition →.py is the only file the AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → may edit. The agent literally cannot win by changing the test.
Speed alone is a corruptible objective, so each run must also clear two gates: perplexity below a threshold and a task-level sanity check. Did the answer contain 48? Did the code prompt return Python? Perplexity alone can't tell you.
One run 'improved' throughput by cutting max tokens to 128 and the model simply did less work. If an agent can raise the score by shrinking the task, you're measuring your ability to fool yourself.
The best keeper wasn't a kernel trick. Deleting about 42 lines of sampler indirection and unused config held identical throughput. Complexity usually arrives with a performance story that doesn't survive measurement.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
Most optimization demos show you the win and hide the search; this one shows the search. An honest walkthrough of applying Karpathy's autoresearch idea to LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → inference — read it for the method and the dead ends, not a tidy result.
Key quotes
“Because if you do not defend quality explicitly, an optimization harness will absolutely "improve" your system by making it worse.”
“If the agent can improve the score by shrinking the task, you are not optimizing inference. You are optimizing your ability to lie to yourself.”
“the hardest part of optimization is not generating ideas. It is building a harness that can tell the difference between a real win, a quality regression, and a benchmark illusion.”
“On the Qwen run, setting sampling to greedy decoding gave the largest gain: about +10.8% generation throughput.”
