Is Memory the Next Scaling Law?

Abstract

Modern AI industry heavily relies on rapid AI capabilities scaling with unsaturated industry benchmarks trending towards exponential growth, which in part offsets enterprises caution towards AI even as most AI pilots fail. This expectation that the scaling law of AI capabilities growing rapidly with model size, data size, and training compute time will hold is one of the key things making at-scale venture investing possible even at current AI pilots success rate. But we’re seeing AI progress achievable in a practical way via pure compute and data scaling slowing down. This suggests that the next breakthrough may be not in doing even more of the same, but in opening yet another scaling dimension, same as when RL and test-time compute diverged from model size growth. This article explores one of such novel dimension that may be memory. We outline a simple token-budget model for measuring memory’s value, discuss design choices for memorization, retrieval, and reintegration, and compare available mechanisms, including retrieval inference integration.

Keywords