Is Weight-Space Programming More Than a Loop Fix?

Abstract

Takehi et al.’s Sparks of Weight-Space Programming offers a tantalizing possibility: give a frontier model access to another model’s weights and let it improve them without gradient descent. We set out to reproduce the reported math improvement, see how far simpler methods could go, and attempt the task with other frontier models. Applying the released edit to Qwen3.5-0.8B raised accuracy from 10.2%to 64.8%on 64 practice problems with four samples each. A difference-of-means script reached 60.9%, recovering 93%of the gain. We supplied the script with Astra’s layers, column budget, and scale; it estimated its own directions and selected its own columns. Those directions had cosine similarities of about 0.78 with Astra’s. Decoding changes also helped, with the strongest reaching 47.7%accuracy. Four blinded runs with Opus 5.5 and Sonnet 5.5 found edits with positive development gains, but stopped after 18–23 forward-pass requests. Different evaluation sets prevent a direct comparison with Astra. We reproduce the practical benefit of the released weight edit and find that a conventional estimator can recover much of it once several important choices are supplied. How much an LLM contributes to discovering those choices remains open.

Keywords