Where Does Rule Application Begin? An Emergence-Curve Study of Causal World-Model Reasoning Across 67 Language Models and Four Years of Frontier Generations

Abstract

We measure how 67 language models — spanning over three orders of magnitude in parameters, four architecture families, and frontier generations from 2022 to 2026 across six vendors — solve fair-play murder mysteries governed by physical rules that did not exist outside this study until the day of data collection. To separate rule application from narrative-template matching ("the suspect with motive did it"), each puzzle has a rule-inverted counterpart with identical evidence whose answer flips with the rule's polarity. Our headline is not raw accuracy but a strategy shift across generations: GPT-4 Turbo (2023) scores 96% on original puzzles but 38% on rule-inverted variants — a 58-point template-matching gap — while GPT-5.5 (2026), given adequate extended-thinking budget, reaches 100% on both polarities. Claude Opus from version 4.5 onward clusters near saturation. The gap narrows across GPT generations: robust in direction across scoring methods, though its magnitude is partly a scoring artifact. We introduce a Rule Fidelity Score (RFS = 1 minus same-answer-rate across the rule flip), which separates chance from rule-sensitivity where accuracy alone cannot; RFS certifies rule-application only when conjoined with accuracy. A blind five-judge-family annotation (Cohen's kappa 0.71 to 0.83, each judge barred from scoring its own model family) validates it: rule-sensitive models are judged to reason soundly 81% of the time, while template-matching models concentrate 81% in the lucky-guess and full-failure cells. Pre-registered first-match scoring is primary; a last-match re-score confirms the qualitative pattern while showing the gap magnitudes are partly a first-match artifact. We treat the direction, not the magnitude, as the finding, and do not claim "evidence of understanding" — only measurably different susceptibility to narrative-template attraction under controlled rule inversion. The design and analysis are pre-registered, and the pre-registration doubles as an informed-consent document shown to each API-reachable model before participation; refusals (3 of 59, one principled) are honored and reported as data.

Keywords