Intel / post

What Happened When I Applied Karpathy's Autoresearch Idea to LLM Inference

Source: x.com
Author: Manthan Gupta
Date: Indexed Apr 15, 2026

Context

Manthan Gupta's experiment shows why a faster generation rate needs an explanation of what changed. His Auto-Inference-Optimiser results report about 10.8% higher generation throughput for a four-bit Qwen 2.5 model with 0.5 billion parameters on an M4 laptop with 48GB of memory.

That experiment replaced randomized sampling with selection of the highest-scoring next . It passed the project's limited quality checks, but removed sampling diversity. The experiment log also rejects an apparent speed gain obtained by shortening the allowed output. The useful result is a measured tradeoff on a specified setup, together with a test process that distinguishes faster execution from doing less work.

Key takeaways · AI-distilled

The splits into two files: prepare.py fixes the model, prompts, warmups and quality gates and is read-only, while inference.py is the only file the may edit. The agent literally cannot win by changing the test.
Speed alone is a corruptible objective, so each run must also clear two gates: perplexity below a threshold and a task-level sanity check. Did the answer contain 48? Did the code prompt return Python? Perplexity alone can't tell you.
One run 'improved' throughput by cutting max tokens to 128 and the model simply did less work. If an agent can raise the score by shrinking the task, you're measuring your ability to fool yourself.
The best keeper wasn't a kernel trick. Deleting about 42 lines of sampler indirection and unused config held identical throughput. Complexity usually arrives with a performance story that doesn't survive measurement.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.

Key quotes

“Because if you do not defend quality explicitly, an optimization harness will absolutely "improve" your system by making it worse.”

“If the agent can improve the score by shrinking the task, you are not optimizing inference. You are optimizing your ability to lie to yourself.”

“the hardest part of optimization is not generating ideas. It is building a harness that can tell the difference between a real win, a quality regression, and a benchmark illusion.”

“On the Qwen run, setting sampling to greedy decoding gave the largest gain: about +10.8% generation throughput.”

Read the source x.com