Can a Linear Program Do Weight-Space Programming?

MIT CSAIL's Sparks of Weight-Space Programming demonstrated that GPT-6 Astra could discern which game a network had played based on its raw weights, generate weights for a small network from a game's rules, and surpass networks trained with PPO for 100 million steps without using gradients or a training loop. They coined this process as weight-space programming.

In another experiment, Astra improved Qwen3.5-0.8B's math abilities by editing only 0.016% of its weights, increasing MATH-500 accuracy from 7.9% to 48.1%. The reasoning model's loops were primarily solved via trial and error, requiring about 8 hours and 28 forward passes. An 8× fewer changed weights edit could achieve the same outcome.

Small reasoning models often get stuck in thinking loops, repeating the same few lines repeatedly. Qwen3.5-0.8B, in particular, was prone to loops, which the model card acknowledged. The MATH-500 training problems faced loops that prevented the model from reaching an answer due to the 32,768-token limit. This weight-space programming fix addressed the loops, benefiting areas where the model was prone to looping, such as instruction following, where IFEval scores improved from 21% to 62%.

On Gemma 3 270M, which rarely looped, Astra couldn't significantly improve its math abilities. Transformers use residual connections, where each layer's output is added to its input rather than replacing it. This model's residual stream is a single vector that every layer adds to, called the residual stream. When the model loops, the residual stream shifts in a consistent direction.

Astra aimed to adjust two layers so that what they added to the residual stream pointed in the opposite direction. Edited columns in two layers were selected based on their contribution to the looping text. By averaging over looping and productive texts, Astra determined the loop direction $u$ and used it to modify the MLP's output.

The edit effectively pushed the model away from looping and improved accuracy on GSM8K and MATH training problems. The entire process was optimized using linear programming (LP) constraints, which minimized the L1 norm of the vector $v$. Astra's full logs and GPU solver (PDHG) were used to determine the optimal values for the edit.