Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights

A new study reveals that large language models (LLMs) can natively read and write neural network weights, exhibiting what researchers call "Weight-Space Programming." This capability may extend beyond expectations from humans or conventional algorithms, offering potential benefits for optimization and interpretability.

For instance, GPT-6 Astra can determine the purpose of a neural network simply by examining its raw weights. Additionally, without any environmental feedback, it can create small game-playing networks from scratch that surpass networks trained using PPO for an extensive 100 million steps.

Furthermore, an LLM named Astra was tasked with enhancing Qwen3.5-0.8B's performance in math without utilizing gradients. The LLM identified an issue in the network's weights causing reasoning loops and fixed this by editing a mere 0.016% of the weights. As a result, MATH-500 improved from 7.9% to 48.1%, and generalization abilities were observed, with IFEval increasing by threefold.

The researchers note that these weight editing capabilities do not require writing training loops, and instead, the LLM uses reasoning to understand the purpose of each part of the neural network and make appropriate modifications. This phenomenon is termed "Weight-Space Programming."

Historically, deep learning relied on symbolic optimizers like Adam or Muon to encode inductive biases. However, this study suggests that LLMs will soon be able to optimize neural network weights natively by directly reasoning about them. The authors compare this to the traditional methods of crafting neural networks for specific tasks like playing Go or predicting their behavior based on raw weights, which are typically challenging for humans.

The study demonstrates that frontier LLMs can now understand and create neural network weights based on natural language goals. By simply providing the weights as input, the LLM can deduce the purpose of the network, identify hidden traits or failures, and even generate entirely new networks from scratch. This paradigm of reasoning-driven optimization expands the potential applications of LLMs beyond text generation and optimization to encompass differentiable neural networks.