Can a Linear Program Do Weight-Space Programming?

Researchers at MIT CSAIL recently reported that GPT-6 Astra could improve a small language model by editing its weights, the numbers that define the model, without any training run (earlier Intel item on the paper). As Nathan Barry summarizes the paper in a follow-up post, Astra changed 0.016% of the weights of Qwen3.5-0.8B, and its score on the MATH-500 math benchmark rose from 7.9% to 48.1%. Barry shows that most of Astra's search for that edit can be replaced by a linear program, a standard optimization problem that an off-the-shelf method solves mechanically. A couple of LP solves of about a minute each did the job of roughly 8 hours of Astra's trial and error, matching its benchmark results within noise while changing 8 times fewer weights (nathan.rs).

The edit fixes one specific failure. This Qwen model tends to get stuck repeating the same few lines, and on MATH-500, 93% of its answers ran into the 32,768-token limit without reaching an answer. Inside a transformer, each layer adds its output to a running vector kept for every token, called the residual stream, and when the model loops that vector shifts in a consistent direction. Astra found that loop direction by comparing the model's internal state on looping and productive versions of the same reasoning, then changed two layers so that what they add points the opposite way. Barry also relays the authors' finding that on Gemma 3 270M, which rarely loops, Astra could not meaningfully improve math, so the edit removes an obstacle rather than adding a skill. Finding the direction took Astra 15 minutes; deciding which parts of the two layers to change, and by how much, took about 8 hours.

Barry's point is that this choice has a clean mathematical form. Each looping token should be pushed away from the loop direction and each productive token barely touched, and both requirements are linear in the amounts chosen, so they become constraints in a linear program. Asking for the smallest total change also tends to make the solver edit only a few components, and Barry kept eight per layer. With Astra's loop direction, the result matched Astra's edit on every benchmark while changing 16,291 weights instead of 130,188. With a direction found by a simple repetition detector, with no LLM involved, it got about 93% of the way.

The analysis also shows what the fix amounts to. The components the LP chose in one of the two layers are active on 89% of tokens and add nearly the same push each time, so the edit works like a constant nudge away from looping. Adding that nudge directly at inference time as a steering vector, a fixed vector added to the residual stream, recovered essentially the whole improvement, but only in a narrow range: at half strength the loops came back, and at double strength they stopped but the reasoning fell apart. Barry tested one model and only the removal of loops, and the LP still depends on Astra to choose the layers, with the best results also using Astra's direction. Within those limits, the steps that still lean on Astra are finding the loop direction, where a simple detector gets most but not all of the way, and choosing which layers to edit; which weights inside those layers to change, and by how much, can be calculated.