Abliteration Behavior Across Four Low-Parameter LLMs: High Rank Subspace Removal Is Generally Detrimental to KL

Abstract

Editing a low-rank direction out of a language model’s residual-stream writes (“abliteration”) involves design choices (which direction, how many, which erasure operator, at which layer) that the literature varies together. We separate them on four instruction-tuned models spanning three architecture families: Qwen2.5-0.5B, OLMo-2-1B, SmolLM2-1.7B, and Falcon-H1-1.5B, a hybrid Mamba/attention stack. The geometry of the refusal direction replicates across all four; every practical ranking built on top of it is model-specific; and the natural ways to measure the comparison are confounded. Replicating 4/4: the first gradient-searched direction recovers about half of the difference-of-means vector and no more (cos=+0.378 to +0.491); Δμ lies mostly outside the span of the found directions (19.7–34.8% of squared norm inside); PC1 becomes the refusal direction in the back half of the stack; the edit folds into the weights exactly; and rank-matched random controls have no measurable effect. Not replicating: the operator ranking takes four different orderings in four models; the style confound spans 4–80%; and “LEACE destroys capability” holds in one model of four. The failure modes are the contribution. Target coupling, ceiling compression and estimator confounding can be shown on one model. Three further failure modes required four. (i) The causal peak for inducing refusal is not the layer at which removing it gives the largest compliance gain: on Falcon the induction peak is layer 11 of 24, but extracting there turns the difference-of-means edit from +0.033 to −0.782 nats of compliance and drops ARC-Easy from 0.893 to 0.800, while the gradient-searched directions change by at most 0.17 nats. (ii) The study’s most reproducible quantity, cos(D 1 ​ ,Δμ), swings further within one model across two layers (0.348–0.491) than it does across four architectures at matched relative depth. (iii) Residual-write coverage is unverified by the standard check: on a hybrid stack the standard module patterns find 24 of 73 write sites, and the fold-versus-hook check still passes, because it only verifies the matrices it edited. One practical result is positive: on XSTest minimal pairs, gradient-searched directions reduce over-refusal to the measurement floor while retaining 80–117% of harmful-versus-harmless discrimination, where difference-of-means retains 35–67%. Harmful prompts are scored only by fixed-string likelihood under intervention; no harmful completion is reproduced.

Keywords