Interpretability-guided weight edits for Gemma 270M, no training
- Source
- balajsaleem.github.io
- Author
- balaj
- Date

Context
An independent researcher reports making Google's small Gemma 3 270M model noticeably better at following instructions by editing a small fraction of its weights directly, with no training run, after AI agents used interpretability tools to find where each instruction got lost. In the author's write-up, the share of held-out test prompts the model followed completely rose from 26.6% to 43.0% while 0.36% of its weights changed. The experiment builds on "Sparks of Weight-Space Programming", a paper from MIT CSAIL in which a large model edits a smaller model's raw weights, the numbers that define it, using nothing but those weights and test runs to check its work (earlier Intel on that paper). The MIT editor found nothing to improve in Gemma 3 270M's math, and the authors concluded that such edits can remove an obstacle hiding an ability but cannot add an ability the model lacks. This post asks whether the editor does better on a blocked ability when it can see what the model is computing.
The "scope" is Gemma Scope 2, a set of published tools that break the model's internal activity into readable features, such as one that fires when the user has asked for a P.S. The first fix shows how the agents used it. Asked to end with a P.S., the model complied 34% of the time on the first try and 86% of the time within 16 tries, so the ability existed but was being blocked. A feature in layer 12 registered the request on all 14 prompts that asked for one, but it was much weaker in replies where the model forgot. Switching off one head, the component that carries information from earlier text forward, dropped P.S. compliance from 43.8% to zero, which marked that head as the route the instruction took. The edit itself added a small amount of the P.S. feature to the table row for the "P", changing 563 weights. Gemma uses that same table to score which token to write next, so "P" gets a boost only when the P.S. feature is active. On prompts the agents had not seen, P.S. compliance rose from 16.7% to 41.7%, and the post reports general-ability test scores were unchanged by that edit.
From there, agents orchestrated by Anthropic's Claude repeated the cycle for six rounds, finding a few recurring kinds of failure, such as keywords that never got copied and instructions like "all lowercase" that faded as replies grew longer. The edits were made using a separate set of diagnosis prompts and scored at the end on 219 held-out prompts from IFEval, an instruction-following test. The author reports a 95% confidence interval of 12.2 to 20.6 points for the overall gain, 99 prompts improved against 25 that got worse, JSON output among the formats that got worse, and general-knowledge scores moving about a point or less. In a direct comparison with the same budget of test runs, the best working from weights alone gained 7.5 points on a different instruction and found nothing for the P.S. instruction, while the best agent with the scope gained 25 points.
The post supports a narrow conclusion: on this model and this task, knowing where an instruction is lost gave the editing agents a direction that searching raw weights did not. The author lists the limits. The work is not peer reviewed, it measures instruction following rather than repeating MIT's math test, and each instruction type has only a handful of test prompts. The scope-guided agents also had about 90 minutes of extra compute for diagnosis, some edits used directions from small probes fitted to the model's internal activity and the diagnosis used gradients, though the model's weights were only ever edited directly, and agents occasionally saw aggregate results from the held-out set. The edited model has not yet been released, though the author plans to release it.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- articleQwen 3.8 27B on my laptop can interpret model weights, toosimonwillison.net
articleThe 'Astra can read and write weights to improve Qwen' post is misleadingsyllogism
article[AINews] LLMs can read & write neural network weights: "Weight Space Programming" cuts Autoresearch out of the RSI loopswyx .ai
Checking sign-in…
Loading comments…