What scale buys for single-cell foundation models: a controlled probe of Geneformer V1, V2-104M, and V2-316M

Abstract

We probe whether Geneformer’s frozen single-cell representations encode known regulatory structure, using donor-stratified linear probes on 100,000 healthy human blood/immune cells (CELLxGENE Census) across three pretrained variants: V1-10M (Genecorpus-30M), V2-104M and V2-316M (both pretrained on ~104M cells). Three probes: cell-type identity (L1), TF→target structure (L2, expression-matched negatives), and hub-TF identity (L3, top-decile out-degree in DoRothEA ∪ CollecTRI). All comparisons include a label-permutation selectivity control (Hewitt and Liang, 2019) and a bag-of-genes baseline. The within-V2 family forms our only clean parameter-scaling axis. Three findings. (1) Hub-TF identity is encoded at the smallest variant (V1-10M, AUC 0.72) and exceeds the bag-of-genes baseline by approximately +0.20 AUC at every scale (V1 +0.22 [+0.13, +0.29], V2-104M +0.21 [+0.14, +0.27], V2-316M +0.20 [+0.13, +0.27]) — a gap that survives a random-initialization control at matched embedding dimensionality (random-init AUC 0.42 at 256d, 0.53 at 768d, 0.44 at 1152d), ruling out the alternative that the signal is geometric room rather than learned structure. (2) TF→target probe gains are large and selectivitypassing across V1→V2 (+0.054 AUC, p=10−16, n=375 TFs) but do not improve under pure parameter scaling within V2 (V2-316M − V2-104M Δ=−0.003 [−0.014, +0.008], p=0.45). The V1→V2 gain is therefore not separable from pretraining-corpus expansion. (3) Cell-type identity (L1) shows no measurable improvement under pure parameter scaling (V2-316M − V2-104M Δ=−0.007 [−0.034, +0.017]); V2-316M fails to exceed the bag-of-genes baseline (Δ=+0.019 [−0.001, +0.043], p=0.31; we report this as “fails to reject,” not “matches,” because we did not perform formal equivalence testing). The contribution is methodological as much as empirical: scaling claims for single-cell foundation models that compare across model-family transitions confound parameter count with pretraining-corpus and vocabulary changes, and the only clean within-family parameter-scaling axis available to us shows no measurable gain on any probe.

Keywords