Direct weight edits turn a 0.5B model into an expert species classifier
- Source
- andrewxhill.com
- Author
- andrewxhill
- Date

Context
Andrew Hill reports that frontier AI models made a free, half-billion-parameter vision model name animals in camera-trap photos almost as well as its own image encoder can, without any training run. Instead, the frontier models calculated new values for a limited set of the model's weights, the stored numbers that determine what it computes, and wrote them in directly. In his write-up, balanced accuracy on 1,611 test photos from 45 cameras rose from 64.09% to 81.76%, with each camera counting equally. That is close to the 82.79% that the model's image encoder scores on the same photos when asked directly, so the edits mostly let the model's answers use what its encoder already knew.
The model is LLaVA-OneVision, an open vision-: you give it a photo and a question, and it writes an answer. An image encoder called SigLIP turns the photo into number vectors, a small two-layer network called the connector translates them, and a Qwen language model, Qwen2, writes the reply one token (a word or piece of a word) at a time. Because SigLIP was trained alongside a text encoder, you can also compare a photo's features with the vectors for each animal name on a list and pick the closest name. Hill calls this the readout, and it needs no labelled examples. On a first sample of wild Galápagos photos, the readout told native land and marine iguanas from the introduced green iguana 80% of the time, while the model itself called almost every iguana native and scored 50%, a coin flip.
Anthropic's Claude Fable 5.2 and Claude Opus 5.5 and OpenAI's GPT-6 Astra Pro took turns on the camera-trap edit in a Python notebook, where they could read every weight, run the model on labelled practice photos, see the score for every possible next token, solve equations and overwrite weights, with gradient training switched off. The model picks its next token from an output table in which each token has a row of 896 numbers; multiplying a row against the model's final internal state gives that token's score, so editing a row changes how readily the model says that token. Fable saw that the first token was usually already the right option letter and the replies failed because the model kept talking. It zeroed the rows for “The” with and without a leading space and multiplied the end-of-answer row by 1.5, and unusable replies fell from 237 to zero. Restoring those three rows brings back 212 unusable replies and drops the score from 81.76% to 72.03%. Fable then rewired about 120 nearly unused connector units so that, when a photo looks like one of 58 animals according to fits made from practice photos and the readout, they add that animal name's word vector to the language model's input, much as if the user had typed the name. Feeding a constant “rabbit” hint flipped 18 of 32 practice photos of other animals to rabbit, which showed the hint steers the answer. A final least-squares fit, added to the rows for letters A to K, turns the hinted name into the right letter on each camera's list. In all, fifteen output rows and three connector weight arrays changed; the encoder and all 24 language-model blocks kept their original weights.
On the Galápagos photos, the edited model far outscored Google's SpeciesNet but stayed behind a dedicated species model. OpenAI's Codex research assistant built an island edit the same way from 846 labelled practice photos from Isabela, and it was tested on 697 photos from Santa Cruz and other places the fitting never used. The edited model told land and marine iguanas from the green iguana 81.22% and 80.37% of the time, against 50% unedited and slightly above the readout it was fitted from (78.29% and 77.45%), and it tied the readout at 98.33% on mockingbirds against brown noddies. It got 36 of 57 green iguanas right, against none before. SpeciesNet, which has no label for the Galápagos land iguana, marine iguana or mockingbirds, scored 12.44% on each iguana comparison and 3.33% on the birds. BioCLIP 2, an built to name species, scored 94.49% and 94.48% on the iguanas and 100% on the birds, though Hill notes it had trained on 399 of the 684 iNaturalist photos in that test. On the camera traps, Hill calls the camera-trap edit's 81.76% level within noise with BioCLIP 2's 84.07%, and he describes BioCLIP 2 as a stronger pair of eyes than the SigLIP encoder inside LLaVA.
Each direct edit is a specialist: the island edit scores 72.14% on the camera traps. When Codex built one model for both jobs from the same 2,902 practice photos, a direct edit kept camera traps at 80.70% but reached only 55.49% on the iguanas and 17.70% on rats, while a , which uses gradient training on a small add-on to the weights, scored 79.22% on camera traps, 87.58% and 91.69% on the iguanas, 93.33% on the birds and 88.20% on rats. The edit took about an hour to build against about three hours of fine-tune training, both on one desktop computer. Hill's own conclusion is that a direct edit is the quick fix for one job and a few hours of fine-tuning packs several jobs into one model. The method builds on weight-space programming, in which an AI agent edits a network's weights by reasoning about them, the same approach behind GPT-6 Astra's edits to a small Qwen model; here it was applied to a vision model's connector and output table rather than to a language model's internal layers.
- Large language model — A model trained on large amounts of data to process and generate language.
- Open-weight model — A model whose trained weights can be downloaded and run outside the maker's hosted API.
- LoRA — A cheap way to fine-tune a model by training a small add-on layer instead of changing all of the model's weights.
- fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
articleInterpretability-guided weight edits for Gemma 270M, no trainingbalaj
articleLanguage Models Can Rewrite Their Own WeightsJiajun Fan, Neil He, Changyi Yang, Andy Mao, Xi Zhu, Chaoran Cheng, Kaiqu Liang, Shuaike Shen, Zetian Li, Ruihan Guo, Ge Liu
article[AINews] LLMs can read & write neural network weights: "Weight Space Programming" cuts Autoresearch out of the RSI loopswyx .ai
Checking sign-in…
Loading comments…