Sparse and Tables Are Enough

Abstract

We present XCP, a language-model architecture that replaces the continuous vector representations of the Transformer with a discrete-symbolic representation combined with sparse linear layers. Tokens are stored as integer IDs (not dense vectors), organized into semantic classes by unsupervised clustering, modeled by discrete class-to-class transition rules, and lexical facts are stored as integer lookup tables; attention is sparsified to 15% of its connections and trained from scratch. On Pride and Prejudice (131k words, ~7k vocabulary), XCP (~0.47M integer parameters) reaches a top-1 test accuracy of 0.841, while a same-scale Transformer (16.6M float parameters) reaches 0.072 even after full training (10,000 steps). Per-layer compression measured: ~1000x embedding storage, ~95% feed-forward fact side, ~85% attention, and ~112x discrete fact lookup. The discrete core is trained by one-pass counting rather than iterative gradient descent. We also report the honest generalization boundary: cross-domain accuracy falls to ~0.11, an information ceiling of the small-data task, not a parameter ceiling.

Keywords