Spectral World Models: Multimodal Operator Learning in Hilbert Space for Imagination-Based Planning

Abstract

Spectral World Models (SWMs) propose a new foundation for agentic world modeling in which environment states are represented as multimodal spectral fields in Hilbert space rather than as Euclidean hidden vectors. Instead of using recurrent state-space models, Kalman-style updates, or transformer attention, SWMs encode text, images, and actions into multiresolution spectral coordinates and evolve them through learned bounded operators. The model predicts the transition law P(S t+1 ​ ∣S t ​ ,a t ​ ), allowing an agent to simulate many possible futures before acting. The framework defines spectral encoders, cross-modal coupling operators, wavelet-based transition kernels, and training objectives for reconstruction, predictive likelihood, and scale consistency. Text and visual information are aligned through shared latent subspaces, enabling language to guide perception and planning. The manuscript connects SWMs to Koopman theory, multiresolution analysis, Mercer operators, and controlled semigroups. It also studies rollout stability, approximation error, computational scaling, and planning behavior. The key claim is that stable, scalable imagination can emerge from spectral evolution of structured multimodal fields rather than token attention or covariance-based filtering.

Keywords