Vibeleaderboard
← All Intel
Intel / article

Language Models Can Rewrite Their Own Weights

Source
arxiv.org
Author
Jiajun Fan, Neil He, Changyi Yang, Andy Mao, Xi Zhu, Chaoran Cheng, Kaiqu Liang, Shuaike Shen, Zetian Li, Ruihan Guo, Ge Liu
Date

Context

A paper from researchers at UIUC, CMU, Rutgers, Princeton and Stanford reports that Qwen3.5-27B, an open model, can edit its own weights to score better on math problems it never saw, with no training run. It does this by reasoning about its own internals and writing the changes as code. Earlier work had shown a frontier model editing the weights of a smaller model; here the model works on itself. The improvement is real but narrow: the edits that counted mostly fixed one failure, answers that run on until they hit a limit.

Each session gives the model its own weights in a Python environment, its graded answers on practice problems, a limited number of runs of itself, and a way to read its hidden states, the internal vectors each layer passes on. Gradients and optimizers are blocked, so every changed value is one the model chose. With a 16,384-token answer limit, the unedited model sometimes reaches the right answer and then repeats a sentence until it runs out of tokens, and 33.6% of its answers on held-out problems ran out of tokens.

In the authors' main protocol, the model's chosen edit is checked once on reserved practice problems before it decides to submit or withdraw. Of the first eight sessions, five withdrew and three submitted. On 490 held-out problems answered eight times each, the best edit raised accuracy from 57.2% to 63.8%, the 6.6 points in the abstract, by making one uniform change to every linear- layer that, in the model's own account, helps long reasoning get stuck less. Runs that hit the limit fell to 26.1%. A second edit gained 4.0 points but made more answers end in repetition, and the third was not significantly better. The abstract's claim that accuracy only rises over eight sessions reflects this setup: withdrawn edits leave the model unchanged.

The repair experiment is cleaner but easier. The authors overwrote eight rows of one weight matrix in a copy of the model with large random noise, which left it solving nothing. Told only that some values were corrupted, the model compared the largest and typical values of every output matrix, found the damaged one, and clipped extreme values. That raised the copy to 33.0% on held-out text problems, against 58.5% for the undamaged model, and on six image benchmarks never used in the session, from 7.3% to 70.2%.

Scale mattered within this one model family. The 2B model made no edit, and the 4B and 9B models added random noise to themselves and broke. Only the 27B model measured before editing. The work shows a capability: a mid-sized open model, in a few hours on one GPU, can diagnose a specific weakness and patch it in its weights. It does not show general self-improvement, since the gains target one failure mode under one on math problems, and other model families and kinds of failure were not tested.

Key takeaways · AI-distilled
  • Self-rewriting needs no pre- or post-training infrastructure, the authors say: the 27B model works from its own weights, graded answers and the ability to run itself, using an open model and hours on one GPU.
  • Given a damaged copy of itself, the model found the corrupted matrix and rewrote it without reading the undamaged weights. The repair, found on text math problems alone, lifted the copy on six unseen image benchmarks from 7% to 70%, 83% of the undamaged model's.
  • The ability appeared only at the largest size tested: across four sizes of one model family, the 2B model made no edit, the 4B and 9B models broke themselves, and only the 27B model improved itself.
  • The 27B model's method compares its hidden states on answers that finish against answers that run out of tokens, then weakens the direction that separates them.
Terms in this piece · Glossary
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • token budget — A cap on how many tokens a task, session, or agent run may consume — the practical control on both cost and how long an agent will grind.
Recommended reads
Comments

Checking sign-in…

Loading comments…