From Single-Model Scaling to Collective Error Scaling: Effects of Training-Experience Sharing, Ensemble Size, and Computational Budgets

Abstract

Scaling studies of neural language models have largely quantified the relationship between model size, training data, training compute, and the predictive loss of a single model. These relationships do not, by themselves, determine how to allocate resources across a system of models. In particular, marginal error rates do not determine the probability that \emph{all} models fail on the same problem. We propose assessing collective error as a separate scaling quantity, together with ensemble size, training-instance overlap, and explicitly defined resource budgets. In controlled synthetic binary-classification experiments using small neural networks, we vary the fraction of exactly shared training examples, $s\in\{0,0.25,0.5,0.75,1\}$, and the number of models, $n\in\{1,2,4,8,16\}$. With a fixed sum of 384 model-update steps, at $n=16$ the average individual error rates for unshared and fully shared training are both close to $6.2\%$, while the all-model error rates are $3.49\%$ and $6.19\%$, respectively. In each of 16 matched synthetic task/seed blocks, the latter is larger. Nevertheless, unshared training does not produce the exponential $p^n$ reduction that statistical independence would predict. Supplementary experiments contrast additional training or width for a single network with multiple smaller networks using an approximate arithmetic-operation budget, an abstention rule, and observed rates of accepted correct and incorrect answers. Differences depend on the budget and the decision protocol but are small. An analytic likelihood-ratio benchmark calculated in the R17 audit shows that, on the synthetic evaluation distribution, the mean of four domain-conditional population-optimal correct-acceptance rates, each constrained to $2\%$ incorrect acceptance, is approximately $85.15\%$. The observed high-budget results are already near this ceiling, limiting conclusions about compute-optimal architectures. The broader question is whether additional AI-development computation should always be concentrated in improving one model, or whether part should instead be allocated to models with distinct learning experience. We do not resolve that optimization problem. We give one controlled reason why individual capability alone cannot settle it: collective failure also depends on the joint distribution of errors. The experiments concern small synthetic tasks and establish neither a scaling exponent for real LLMs nor an optimal design for scientific research sys

Keywords