Truth Directions: Signal-to-Noise and Geometry of Recoverability
Introduction
The linear representation hypothesis states that a model encodes high-level concepts as directions in activation space. This reduces reading a concept to projecting the residual stream along the concept direction, and steering to adding a multiple of the concept vector to the residual stream. Marks & Tegmark (2023) showed that in sufficiently large models, a linear direction corresponding to an abstract notion of truth emerges and applies across diverse sets of inputs. They estimated this direction with a difference-in-means (“mass-mean”) direction fit on true/false statements and showed that it separates held-out statements, transfers across datasets, and is causally implicated under intervention.
Previously, Burns et al. (2022) had proposed finding a linear truth direction without labels, by demanding logical consistency, a method they call Contrast-Consistent Search (CCS). However, Roger (2023) showed empirically that the CCS estimator is insufficiently constrained and showed that untrained, randomly initialized probes could already reach about $75\%$ accuracy on the “easy” dataset once the CCS convention of inverting the sign of a below-chance probe was applied. Additionally, in his study more than twenty mutually orthogonal “truth probes” reached accuracies similar to the one returned by the CCS estimator, implying that CCS has not found the optimal linear probe. Moreover, due to the inversion convention in choosing the sign, comparing to a random baseline is misleading because the accuracy of this baseline will always be either greater than or equal to one half. Farquhar et al. (2023) studied the CCS estimator further and showed that because arbitrary binary features are optimal under the consistency loss, nothing in it selects for knowledge, and in practice unsupervised probes recover whatever feature is most prominent in the activation space representation. This implies that when using unsupervised methods to obtain a linear truth direction, the linear direction obtained via the estimator can appear to decode truth while actually corresponding to a salient direction that happens to correlate with the label on that particular dataset. These observations suggest that the decoding of truth inherits a signal-versus-noise problem. Consequently, on a benchmark where chance-level structure is very strong, the ability of a truth probe to separate the data into true and false classes is a much weaker result than it first appears. This post applies the same signal-to-noise analysis, raised by critiques of the CCS estimator, to linear probes fit from labeled statements.
A mass-mean “truth direction” is an estimator for a causal truth direction which separates true and false statements. However, in the low signal-to-noise regime, the estimator can end up returning a direction with a large component lying along the most salient direction in activation space, i.e. the direction of largest within-class variance. On a dataset where the class gap is small relative to the within-class spread along the axis connecting the centroids, the finite-sample mean difference is dominated by noise along that axis, and the estimator contains a large component along the salient axis regardless of whether the salient axis has a large overlap with the recoverable truth direction. This implies that a truth direction and a salient axis can look alike on a benchmark and require a signal-to-noise analysis to distinguish them. In this post, I apply a signal-to-noise analysis to mass-mean truth estimators to address the question of recoverability: when does a model contain a linear truth direction that can be recovered by a probe that carries signal beyond the single most salient axis?
Systems neuroscience has spent decades asking what a downstream reader can recover from a population of noisy neural units. In this spirit, I treat probing for a truth direction as a readout problem: I take the probe as a linear readout, quantify its separation with a detection-theoretic $d'$, and benchmark that $d'$ against an explicit random-direction null across the Pythia scale ladder on twelve benchmark datasets. This yields four results:
(1) Apparent separation is trivial. If the size of the dataset puts the model below Cover’s capacity ($N \ll 2d$ throughout) for a binary classifier, then it is possible to find a direction which can separate any random relabeling of the data for points in general position, which is generically true for activations. This causes in-sample separation to be inflated and it must be fixed by scoring on held-out data not used to fit the direction. Additionally, a random direction inherits a share of the real class gap between the true and false centroids by virtue of projection, i.e. $u^{\top}\delta \sim \mathcal{N}(0, \lVert\delta\rVert^2/d)$. This implies that the performance of the fitted probe relative to a random null direction is the relevant quantity, not the raw score of the probe.
(2) When the class signal is weak, the estimator returns a nuisance direction. In this case the mass-mean estimator returns the dominant activation axis $\hat v_1$ rather than a truth direction. Recoverability for the mass-mean probe is determined by whether the class gap grows with depth relative to the spread such that the held-out AUROC clears the random-direction null. On the cities dataset, the spread begins to shrink and the gap begins to grow at layer 11, and by layer 14 the held-out AUROC clears the null for the remaining layers of the network. However, on counterfact, the class gap never dominates the spread, which only collapses in the last two layers of the network where the separation is still too weak to provide a usable readout. The decodability results replicate on OLMo-2-1B across a different architecture and corpus.
(3) When the estimator returns a nuisance direction, steering along it fails to correct the behavior. Because the estimator has failed to return the causal truth direction, steering along the estimator does not generally increase the score for true completion and can in some cases provide a lower score when compared to steering along random directions for displacements of the same magnitude. All steering measurements are done on Pythia.
(4) The failure is in the mass-mean estimator and not in the absence of a causal truth direction: the nuisance direction corresponding to the salient axis is identified in advance from the within-class spectrum and the alignment of the fitted direction with the salient direction. Correcting the estimator via whitening raises the held-out AUROC above the random decoding null and returns the correct steering behavior in counterfact.
Activations and class statistics
For each experiment, I start by fixing a model and dataset. Each dataset consists of $N$ pairs of labeled statements $(x_{i}, y_i)$ with the truth labels $y \in \{0, 1\}$. For each statement, I get the residual stream activations by passing it through the model once and summarizing it by the residual-stream activation at its final token for each layer, i.e. $x_{i,L} \in \mathbb{R}^{d}$, where $d$ is the model’s hidden width, $i$ the statement index, and $L$ the corresponding residual stream layer.
Throughout the post, a hat denotes a quantity estimated from a finite sample, with population expectations taken over the data-generating distribution of activations at the probed layer. A table of every symbol used in the post is collected in the notation appendix. The distinction between sample and population estimates is crucial: almost every issue studied in this post can be traced back to the gap between an estimate and its population target, $\hat\theta$ and $\theta$, or the eigenvalues of $\hat C$ and of $\Sigma$. I fit quantities on a training split of $N_{\text{train}} = N_0 + N_1$ activations, $N_0$ of them with $y = 0$ and $N_1$ with $y = 1$. Held-out data is never used to fit a direction, I only use it to evaluate the held-out performance (AUROC) of the probe.
I define the population class means and their estimators as
I define the class-mean gap as $\delta = \mu_1 - \mu_0$, with $\hat\delta = \hat\mu_1 - \hat\mu_0$, and the within-class covariance as the class-weighted average of the two class-conditional covariances,
The estimator for $\Sigma$ centers each class on its own mean, so that the between-class shift does not leak into the noise estimate:
with $\hat C_0$ and $\hat C_1$ the sample covariance of each class (dividing by $N_0$ and $N_1$). For balanced classes $\pi_0 = \pi_1 = \tfrac{1}{2}$, and $\Sigma$ reduces to the plain average $\tfrac{1}{2}\big(\operatorname{Cov}[x\mid y{=}0] + \operatorname{Cov}[x\mid y{=}1]\big)$.
These population quantities have no closed form: they are the $N_{\text{train}}\to\infty$ limits under the data-generating distribution and are never obtained directly. Their influence is visible only through held-out performance against the random-direction null. The one exception is the whitening schematic below, whose activations are drawn from a known Gaussian, so there $\Sigma$ and the optimal direction $\Sigma^{-1}\delta$ are known by construction rather than estimated.
Probes, estimators and scores
Probe. A probe is defined as a unit vector $u \in \mathbb{R}^{d}$, $\lVert u \rVert = 1$, that reads the scalar $z = u^{\top} x$.
Steering. Steering is the inference-time counterpart of reading. To steer along a concept, one adds a scalar multiple of the corresponding vector to the residual stream at a chosen layer of the model during the forward pass, i.e. $x \mapsto x + h\,c\,w$, and measures how the model’s output moves. Steering along a truth direction is expected to carry the representation of a statement to a point in the residual stream where the model favors the true completion. This is tested relative to a steering null described in Drawing a random direction.
Generic steering directions are written $w$ and generic readout directions $u$. The specific directions I consider carry their own names. The unit $c$ and coefficient $h$ are fixed in The scale of an intervention below.
Mass-mean probes from the difference-in-means direction. The mass-mean direction is defined as
i.e., the unit vector from the false-class centroid to the true-class centroid. It fits two averages and thresholds $z$ at the midpoint $\hat b = \tfrac{1}{2}(\hat\mu_1 + \hat\mu_0)^{\top}\hat\theta$.
Fisher (whitened) direction. The Fisher direction is defined as
and is normalized to unit length. The multiplication of $\hat\delta$ by $\hat\Sigma^{-1}$ downweights components of the mean shift that lie along high-variance nuisance directions. Here $\hat\Sigma$ is the Ledoit–Wolf shrinkage estimate $\hat\Sigma = (1-\rho)\,\hat C + \rho\,\frac{\operatorname{tr}\hat C}{d} I$, with $\hat C$ the within-class estimate above and $\rho \in [0,1]$ the closed-form Ledoit–Wolf intensity. Shrinkage is needed because the raw $\hat C$ is singular whenever $N_{\text{train}} < d$, the relevant regime here. The Ledoit–Wolf intensity is chosen to minimize the error of $\hat\Sigma$ itself, not to maximize $d'$ of the whitened direction, which makes every whitened number in this post a lower bound as described in the methods appendix.
Separation, $d'$. For a probe direction $u$, one can project both class-conditional clouds onto $u$: each collapses to one dimension, with means $u^{\top}\mu_0$, $u^{\top}\mu_1$ and variances $u^{\top}\operatorname{Cov}[x\mid y{=}0]\,u$, $u^{\top}\operatorname{Cov}[x\mid y{=}1]\,u$. The separation $d'(u)$ is the class-mean gap projected along $u$, in units of the projected within-class spread:
The numerator is the projected class-mean gap, the denominator the projected within-class variance, which for balanced classes is the pooled variance $\tfrac{1}{2}\big(u^{\top}\operatorname{Cov}[x\mid y{=}1]\,u + u^{\top}\operatorname{Cov}[x\mid y{=}0]\,u\big)$, since $\Sigma$ is the class-weighted average of the two. Writing it this way makes the object a Rayleigh quotient in $u$ from the start, which is the form every later result takes.
Thus, $d'$ is the separation measured in units of its own noise, the detection-theoretic sensitivity of an ideal observer discriminating two Gaussians. Defining the square first makes the sign-blindness explicit rather than asserted: $d'$ depends on the mean difference only through $(u^{\top}\delta)^2$, so it cannot distinguish a direction from its negation.
I subscript throughout:
for the mass-mean, whitened and projection-out directions defined below. The optimal $d'$ over directions is the Mahalanobis separation. The Rayleigh quotient in $u$ is maximized at $u \propto \Sigma^{-1}\delta$, which is the Fisher direction defined above, and its maximum value is
so the whitened direction is the solution of the above optimization of $d'$, and the mass-mean direction only coincides with it when $\Sigma$ is a multiple of the identity. A held-out $d'$ used to characterize the separability of a fitted direction replaces $\delta$ and $\Sigma$ by their sample values on the remaining data not used to fit the direction.
AUROC (area under the receiver-operating-characteristic curve). The AUROC is the probability that a randomly chosen true statement projects above a randomly chosen false one, $\mathrm{AUROC}(u) = \Pr[\,z_1 > z_0\,]$ for $z_1 \sim p(z\mid y{=}1)$, $z_0 \sim p(z\mid y{=}0)$ independent. Unlike $d'$, it is sign-aware: $\mathrm{AUROC}(-u) = 1 - \mathrm{AUROC}(u)$. The name is a radar-era inheritance and the quantity is described further in the appendix AUROC and its relation to $d'$.
$\mathrm{AUROC}$ and $d'$ are linked when the class-conditionals are Gaussian and the classes balanced, so that $\Sigma$ is the plain average of the two class covariances:
with $\Phi$ the standard normal CDF. So $d' = 1$ corresponds to
$\mathrm{AUROC} = \Phi(0.71) \approx 0.76$, and $d' = 3$ to $\approx 0.98$. Equal class
variances are not required: for balanced classes the variance of $z_1 - z_0$ is
$2\,u^{\top}\Sigma u$ whether or not the two class covariances agree. Away from the
Gaussian case the identity is only a guide, which is why both are reported. On the data
here it holds to within $3\%$ under shuffled labels for cities, larger_than and sp_en_trans even where the projections carry excess kurtosis above $1$, as Dimensional slack and sample size reports. This is described in Appendix: The effective-dimension prefactor. It also holds for counterfact but only after removing the statements which drop the massive activations. These statements are the subject of The origin of the rogue dimension.
Separability, capacity and readout
The mass-mean direction. Geometrically, obtaining $\hat\theta$ is the most naive thing one could do: project onto the line joining the two class centroids. There is no fitting beyond two averages, so there is little room to launder a spurious feature in through an optimizer. Therefore, whatever separation it finds is a property of the data, not of an optimizer’s freedom to search.
Cover’s theorem: the counting baseline. Before asking how well a direction separates the classes, first one must ask how surprising it is that a separating direction exists at all. Cover’s function-counting theorem (1965) is an exact result that states that for $N$ points in general position in $\mathbb{R}^{d}$, the number of the $2^{N}$ possible binary labelings that a hyperplane through the origin can separate is
where general position is defined by every subset of at most $d$ points being linearly independent which activations satisfy generically. The fraction of labelings that are linearly separable is then $f(N,d) = C(N,d)/2^{N}$ (a probe with a bias term is an affine hyperplane, the homogeneous case in one higher dimension). The fraction stays near $1$ for $N \lesssim d$ and falls through $\tfrac{1}{2}$ at the separating capacity $N = 2d$. Since $d_{\text{model}}$ runs from a few hundred to a few thousand across the ladder, and each probing set is hundreds to low thousands, probing sits at or below that capacity. This means that a large fraction of labelings are linearly separable so the linear separability of the truth dichotomy provides little information. It reflects the ambient dimension more than anything the model has learned. The question is therefore never whether a separating direction exists, but how far above the random null the particular mass-mean direction lands.
Cover’s theorem suggests that $N/2d$ is the natural unit for the empirical analysis, with $N \lesssim 2d$ being the regime in which the dimensional slack is available to any fit. The shuffled-label control asks how much of the dimensional slack the mass-mean rule converts into apparent signal when the labels are random and thus carry no signal at all.
The random-direction null. Roger’s and Farquhar’s results show that a nonzero $d'$ is not, by itself, evidence of anything. The meaningful quantity is therefore not $d'$ obtained from the estimator but $d'$ relative to that obtained by the random null control. This is obtained by drawing many random unit directions $u \sim \mathrm{Unif}(S^{d-1})$, forming the distribution of their $d'$ (equivalently AUROC), and taking its 95th percentile,
A truth direction is recoverable only insofar as its own score clears that quantile, i.e. $\mathrm{AUROC}(\hat\theta) > p_{95}$. This is the observation in Roger’s post turned into a control, first done by Mallen & Belrose (2023), resolving each random probe’s sign on its source distribution and scoring transfer against quantiles of $10^7$ random-probe AUROCs. The per-layer null, the $d'$ scale, and the margin criterion are added by this post. The report that random probes reach ~75% on the “easy” datasets now becomes a chance level measured for each model, layer, and dataset. Every claim below is stated relative to this measure.
For a direction $u$ evaluated on held-out data against the null of its own layer and dataset, the margin $m(u)$ is given by
with $p_{95}$ as defined above, computed on the same held-out points. A direction counts as recovered when $m > 0$, a fixed 5% false-positive rate against random directions. Layer selection maximizes $m$ over depth, $L^{*} = \arg\max_{L} m(u_L)$, each layer scored against its own null, not the raw AUROC. The same construction on the $d'$ scale gives the equivalent criterion: a random direction’s sign is arbitrary and $\mathrm{AUROC}(-u) = 1 - \mathrm{AUROC}(u)$, so each null draw’s AUROC is folded to $\max\{a, 1-a\} \ge \tfrac{1}{2}$, and the folded AUROC is monotone in the sign-blind $d'$ through the Gaussian link above.
Whitening: the Fisher direction. The mass-mean probe ignores the shape of the within-class noise. If the class-conditional covariance $\Sigma$ is anisotropic, with large nuisance variance along some directions, the optimal linear discriminant is not $\mu_1-\mu_0$ but the Fisher/LDA direction
This correction is not introduced here. It is the second half of mass-mean probing itself. Marks & Tegmark set $\theta_{\mathrm{mm}} = \mu_1 - \mu_0$ and then, for IID evaluation, read with $\sigma(\theta_{\mathrm{mm}}^{\top}\Sigma^{-1}x)$, noting that this coincides with linear discriminant analysis. Their Appendix E gives the same construction in terms of Mahalanobis whitening, and their $\Sigma$ is the within-class covariance defined exactly as above. The issue is where the correction is applied. They are explicit that $\Sigma^{-1}$ is there to tilt the decision boundary, while $\theta_{\mathrm{mm}}$ remains the candidate feature direction, one which may be non-orthogonal to that boundary. Their intervention experiments steer along $\theta_{\mathrm{mm}}$. In their framework, the whitened vector is a readout and the raw mean-difference is the feature. The steering results below report a regime in which that assignment is inverted.
Whitening should raise $d'$ when the signal is partly buried under structured noise, and hurt when $n \ll d$ and $\hat\Sigma$ is ill-conditioned, i.e. where the inverse amplifies estimation error. I estimate $\Sigma$ with Ledoit–Wolf shrinkage for exactly that regime, and report plain and whitened side by side, since the gap between them is itself diagnostic of how anisotropic the truth geometry is.
The readout framing puts this in familiar company: whitened comparisons of representational geometry have become standard in work on artificial networks (Diedrichsen et al., 2021), and Ying et al. (2026) measure a Mahalanobis cosine between truth probes for exactly the reason above: these representations are anisotropic enough that a raw inner product misleads. A follow-up by Ying, Hase & Kriegeskorte (2026) proves the link between that cosine and readout quality in closed form. For balanced classes with Gaussian projections, a probe’s held-out AUROC is $\Phi(s/\sqrt{2})$ in its signal-to-noise ratio $s$, and its Mahalanobis cosine to the Fisher direction is a softsign in the same $s$. Their $s$ is the $d'(u)$ defined in this post computed with the pooled within-class covariance, and their Fisher distance is $d'_{\mathrm M}$. One of the conditions under which they show the law fails, a difference-of-means probe far from the Fisher direction, is the regime studied below.
Superposition: the salience knob. Since features share the residual stream, the directions of largest variance need not be the directions of interest. The dominant principal components may encode whatever is most prominent on this dataset (sentence length, topic, token identity) rather than truth. Here a salient direction means a leading principal component of the activations, i.e. one of the directions of largest variance. This is Farquhar’s “most prominent feature” stated geometrically, suggesting a direct test: fit the mass-mean direction after projecting out the top-$k$ principal components of the activations, taken on the total covariance and so including any between-class shift, and analyze $d'$ as a function of $k$. If truth is merely contained in the salient subspace, the signal collapses as soon as the leading components are removed. However, if it occupies its own low-variance subspace, $d'$ survives.
Drawing a random direction
I build both nulls in this post from random unit vectors. As mentioned in the previous section, to sample $u$ uniformly from the unit sphere $S^{d-1}$, I draw a standard Gaussian and normalize:
This is uniform because the isotropic Gaussian density depends on $\xi$ only through $\lVert \xi \rVert$, so it is invariant under every rotation. Normalizing leaves a distribution on the sphere with the same invariance, and the uniform distribution is the only one with that property. (Normalizing each coordinate independently, or sampling coordinates uniformly in $[-1,1]$ and normalizing, does not give a uniform direction, since it concentrates near the cube’s diagonals.)
Two facts about Gaussian random vectors in high dimension enter here. The overlap of such a vector $u$ with any fixed unit vector $v$ is small, $u^{\top} v \sim \mathcal{N}(0, 1/d)$ to a good approximation, so typically $\lvert u^{\top} v\rvert \approx 1/\sqrt{d}$. Likewise, the overlap with the coordinate axes is $O(1/\sqrt{d})$. This implies that a random direction is nearly orthogonal to any direction fixed in advance, and by a union bound to every direction on a list at once, so long as the list is not exponentially long in $d$.
The decoding null. A random direction does not achieve $\mathrm{AUROC} = \tfrac{1}{2}$ merely up to sampling noise. Its projection inherits a share of the real mean shift: $u^{\top}\delta \sim \mathcal{N}(0, \lVert\delta\rVert^2/d)$, so
When $\Sigma$ is close to isotropic, $u^{\top}\Sigma u \approx \operatorname{tr}\Sigma / d$ and the factors of $d$ cancel, leaving $d'(u) \approx \lvert z \rvert \cdot \lVert\delta\rVert / \sqrt{\operatorname{tr}\Sigma}$ with $z \sim \mathcal{N}(0,1)$. The null is therefore large exactly when the true separation is large. It is not a fixed background, and it must be recomputed for every layer and dataset.
The steering null. A steering direction $w$ is added to the activation rather than projected onto it. I draw $w$ the same way, add it to the residual stream, and measure how the model’s behavior moves: this measures what an arbitrary direction causes, rather than what it reads. A direction can clear the decoding null without clearing the steering null or vice versa. Throughout this post an effect, decoding or steering, is called significant when it clears the 95th percentile of its null, and the word is used in no other sense.
The scale of an intervention
The behavioral score. Steering moves the model’s preference between two
completions of the same prompt. Each statement is split at its final entity into a
prompt and a pair of one-entity completions. On cities the prompt is “The city of
Krasnodar is in” with true completion “ Russia” against a paired false country. On
counterfact, the shipped relation template, e.g. “.NET Framework is created by” with
“ Microsoft” against “ Google”. The score is
summed over the completion’s tokens. Both completions are scored against the same prompt, so the prompt’s own likelihood cancels and only the preference between the two entities remains.
Steering displaces the residual stream, $x \mapsto x + h\, c\, w$. The unit $c$ sets the scale, and should be chosen so that the effects of interest appear at $h \sim 1$. Much smaller displacements cannot move a statement across the decision boundary, and much larger ones push the activation far outside the range the model ever sees, where the response is large, idiosyncratic, and mostly independent of which direction was pushed. The steering null widens sharply in that regime, which is a useful diagnostic in its own right. If a random direction of the same length produces as much behavioral change as the chosen direction, $h$ is too large.
Taking $c = \sigma_w = \operatorname{std}(w^{\top} x)$, the spread of activations along $w$, seems natural but is a trap: $\sigma_w$ depends on the direction, so different conditions of an experiment receive pushes of different magnitude. It also depends on the layer, so a fixed $h$ means different things at different depths.
The right unit is the class-mean gap $c = \lVert\hat\delta\rVert$, which is a property of the dataset and layer rather than of the direction. Then $h = 1$ displaces an activation by exactly the distance between the two class means, so that along $\hat\theta$ it carries the false-class centroid onto the true-class centroid. Every displacement is norm-matched and each layer is comparable.
Decomposition. The shift in $\ell$ at coefficient $h$, averaged over the evaluation set, is $\Delta(h) = \big\langle\, \ell(x + h\, c\, w) - \ell(x) \,\big\rangle$. I express the symmetric and antisymmetric parts of the shift as:
Only the antisymmetric part $A$ is evidence of a direction: pushing along $\hat\theta$ should make the model favor the true completion and pushing against it the false one, so a genuine truth direction responds as an odd function of $h$. A generic perturbation degrades the computation whichever way it points, contributing to $S$ and leaving $A$ near zero.
Seeds and draws. I define a seed as a resampling of the class-stratified train/test split. Averaging over seeds gives the standard error on the given direction’s effect, that is, the reproducibility of the measurement. I define a draw as a resampling of a fresh random direction ($u$ for a decoding null, $w$ for a steering null). Since no fitting is involved, the drawn direction does not depend on the split, and the draws simply provide the null distribution. A tight seed error bar says nothing about whether the effect sits inside the null.
Empirical setup
Models. I use the Pythia suite (Biderman et al., 2023) at 70m, 410m, 1.4b, and 2.8b parameters. They share architecture, data and training order and differ only in scale, which is what makes it the right ladder for an emergence question. Residual widths are $d = 512, 1024, 2048, 2560$.
Data. I use the twelve curated true/false sets from
Marks & Tegmark. Nine form a
main tier, cities, neg_cities, larger_than, smaller_than,
cities_cities_conj, cities_cities_disj, common_claim_true_false,
companies_true_false and counterfact_true_false, subsampled class-balanced to
$N = 1198$, the size of the smallest member rounded to a class-balanced count, so that $d'$, transfer, and the ratio
$N/2d$ are directly comparable across datasets. Two translation sets
(sp_en_trans, neg_sp_en_trans) have only $N = 354$ and are reported separately. The likely set is included as a
distractor: it is constructed so that textual plausibility decorrelates from
truth, so a direction fitted on it isolates the plausibility axis on its own. These
datasets, the papers that introduced them, and their known failure modes are catalogued
in a companion map of the truth-probing
literature.
Activations. I pass each statement through the model once, and the residual stream is read at the final real token (right-padded, located from the attention mask) at every layer. Layer $0$ is the embedding, before any transformer block. On templated statements the last-token embedding is nearly constant within a dataset, so its $d'$ is degenerate and it is excluded from layer selection.
Estimation, held out. I fit every quantity on a class-stratified training half and evaluate on the held-out half: the mass-mean direction, the within-class covariance used for whitening, and the PCA basis used by the superposition probe. The estimator is not prone to overfitting in the usual sense, since it has almost no capacity, fitting only two class means. However, with $d$ comparable to $N$ the estimated mean difference absorbs $O(\sqrt{d/N})$ of noise, and scoring it in-sample biases $d'$ upward through exactly the fluctuations that defined the direction. In practice the correction is large. On a synthetic control with isotropic noise, a planted separation of $d' = 1$, and the post’s own $d = 2560$ and $N = 1198$, the in-sample estimate returns $4.2$ and the held-out estimate $0.24$. Both follow from the derivation in Dimensional slack and sample size: the noise in $\hat\theta$ is aligned with the sample fluctuations that produced it, so in-sample $d'^{2} \simeq 1 + 4d/N_{\text{train}}$, while held-out $d' \simeq \cos(\hat\theta, \theta) \simeq (1 + 4d/N_{\text{train}})^{-1/2}$. The simulation matches the in-sample prediction to within $2\%$ and the overlap $\cos(\hat\theta, \theta)$ to within $4\%$ for $d/N_{\text{train}}$ from $0.2$ to $14$. The held-out $d'$ itself scatters more, $-11\%$ to $+16\%$ about its prediction, because it is estimated on a finite held-out half: its standard deviation across repetitions is about $0.08$, so a twenty-repetition mean is fixed only to about $\pm 0.02$, and every deviation sits within two of those.
Two consequences follow. First, the held-out $d'$ is attenuated: the fitted $\hat\theta$ is misaligned with the true direction by $O(\sqrt{d/N_{\text{train}}})$, so the reported $d'$ underestimates the intrinsic separation on average, and the attenuation is largest for the widest model. Second, $d'$ is sign-blind while AUROC is not, so $\hat\theta$ is oriented on the training half and that sign is held fixed on the test half. Layer selection then maximizes held-out AUROC above the layer’s own null, which cannot reward an anti-predictive direction. Selecting a layer by an SNR is itself the field’s practice. Bürger et al. take the layer maximizing the ratio of between- to within-class variance, and later work adopts the recipe unchanged (Bao et al. select the layer for every model in their study by that ratio, crediting Bürger et al. and MacDiarmid et al. for the technique. Poulis et al. App. B.1 write out the form Bürger and Bao are using, a ratio of squared mean differences to variances averaged across dimensions). That is $d'^{2}$ averaged over coordinates, with no direction and no null defined. The margin used here is the same ratio along one direction, read against what random directions give.
The null. At every layer I draw $200$ random unit directions for the decoding null, evaluate them on the same held-out points, and fold as above.
Whitening. $\hat\Sigma$ as defined above. At $N_{\text{train}} \approx 600$ against $d = 2560$ the shrinkage is what makes the inverse usable.
Regime of recoverability
With the null, $d'$, and the class-mean-gap unit defined, at what scale, at what depth, and across which datasets is a truth direction actually recoverable?
For both cities and counterfact the held-out AUROC at the best layer shows that recovery occurs starting at pythia-410m. For cities, the best-layer recovery can be done via mass-mean probe or whitened probe while for counterfact recovery requires a whitened probe for pythia-410m.
The direction transfers, but only within a polarity and a family. Fitting the
mass-mean direction on one main-tier set and evaluating it on another reproduces the
within-dataset signal on the diagonal, $0.93$ to $0.98$ for the single-frame sets and
down to $0.55$ for counterfact, but the off-diagonal is organized, not merely weak.
Three structures are visible in the full nine-by-nine matrix. First, between a
statement type and its logical negation the transfer is anti-predictive: cities
scores AUROC $0.08$ on neg_cities and larger_than scores $0.07$ on
smaller_than, both far below chance, so the same vector reads truth backwards once
polarity flips. The neg_cities direction is anti-predictive not on cities alone but
on the compound and claim-like sets as well, $0.02$ to $0.33$ across five of them.
Second, the sets fall into two families that do not speak to each other. The numeric
comparisons transfer to nothing outside their own pair, $0.34$ to $0.63$ in both
directions, while cities, the two compound sets, common_claim, companies and
counterfact form a block that transfers within itself at $0.52$ to $0.94$. Third,
that block is asymmetric, and the asymmetry belongs to the receiver rather than to the
direction. The cities column is uniformly high, $0.77$ to $0.94$ from every other
member, because a transferred score is $\Phi(d'/\sqrt2)$ with the target’s gap and
noise inside $d'$, and cities has the largest gap in the benchmark. A direction that
reads its dataset at $0.55$ still reads cities at $0.77$. Transfer AUROC scores the
receiver’s signal-to-noise as much as the direction’s alignment, so the matrix has to
be read by rows and columns together. The mass-mean direction is genuinely there, but a
single vector approximates a structure carrying at least a separate polarity axis and
a family axis. The below-chance cells are what the decomposition of
Bürger et al. (2024) predicts: a direction fit on
affirmative statements alone is a mixture $t_A = \alpha\, t_G + \beta\, t_P$ of a
general truth direction and a polarity-sensitive one that reads truth forwards on
affirmatives and backwards on negations, so under negation the $t_P$ term
anti-correlates with the label, and an AUROC of $0.08$ rather than $\tfrac{1}{2}$
says that term dominates the mixture at this layer. The comparison is direct, not
analogical: their cities/neg_cities pair is the Marks & Tegmark pair used here.
Poulis et al. (2026) measure the same signature
in Llama, affirmative-trained probes anti-predictive on negations
(AUROC $\approx 0$) at the depths where the polarity direction carries most of the
truth-related variance.
The sweep covers all twelve sets, not only the nine of the transfer matrix. Every
templated set with a single frame clears its null comfortably (plain AUROC
$0.93$–$0.98$ at its selected layer). companies_true_false
shows the largest plain/whitened gap in the benchmark after counterfact, a second instance of the
rogue-dimension pattern, decoded rather than steered. The full table and
per-dataset commentary are in the appendix Results on the full twelve-dataset benchmark.
The plausibility axis has the opposite depth profile. The likely set carries no
truth labels at all. Its two classes are the most likely and hundredth-most-likely
final token, so a direction fitted on it reads textual probability alone.
This axis is easy to find. On pythia-2.8b the mass-mean direction reaches held-out
$\mathrm{AUROC} = 0.890$ ($d'_{\mathrm{mm}} = 1.75$) at layer $12$, against a null 95th percentile of
$0.593$, a wider margin than counterfact achieves at any depth. What separates it
from the truth sets is the depth at which it is recoverable. Plausibility is already
readable in the embedding ($0.700$ at layer $0$), peaks at layers $11$ and $12$ (the AUROC at $11$, the margin at $12$), and then
decays through the second half of the network to $0.657$ by layer $28$. cities
runs the other way: inside the null through the early layers, clear of it by
mid-network, peaking at $0.973$ at layer $29$.
Marks & Tegmark designed the likely dataset for a cross-dataset test: fit a direction on a truth dataset and evaluate it on likely, where plausibility is decorrelated from truth. A direction that had been reading plausibility all along should separate likely. However, none of the fitted directions achieve this. Across all nine main-tier sets, a direction fitted at that set’s own
selected layer and evaluated on held-out likely gives AUROC between $0.41$ and
$0.60$, and only counterfact clears the random-direction null on likely at the corresponding
layer ($p_{95} \approx 0.58$–$0.60$), and only after sign-folding. The largest is larger_than at $0.598$ against a
null of $0.600$. cities gives $0.559$ and counterfact $0.406$. However, since the likely null is sign-folded, the held-out AUROC of counterfact folds to $0.594$ against $p_{95} \approx 0.584$ at layer 31, indicating that the counterfact direction marginally separates likely, but it does so backwards. The reverse direction agrees: the likely-fitted direction scores $0.43$–$0.54$ on the nine truth sets. The plausibility axis and the truth directions are therefore separately readable and nearly mutually uninformative. The depth profile of truth and plausibility are cleanest at 2.8b so this is where using plausibility as a distractor is most useful as a test
Dimensional slack and sample size
Cover’s theorem marks $2d$ as the scale below which separability stops being informative. The shuffled-label control measures what that costs in practice: fit the mass-mean direction to a random permutation of the labels and record how much separation comes back.
The control is the control task of
Hewitt & Liang (2019), and the gap between
real-task and control-task performance is what they call selectivity. Their diagnosis
and their prescription concern probe capacity: a probe expressive enough to memorize
the control task is too expressive to trust, so a smaller one should be used. However, the mass-mean probe only fits the two class means, and on counterfact at $N = 100$ it still separates shuffled labels with AUROC
$0.83$ on pythia-2.8b ($0.825 \pm 0.067$ over sixteen permutations), which is close to what
the true labels reach in sample at the same $N$. Since the mass-mean probe only fits two parameters, the freedom is in the ambient dimension.
This control is the one place in the post where a direction is scored on the points it was fitted to. Everywhere else a direction is fitted on a training split and scored held-out. Here that would defeat the purpose, since on held-out data a shuffled-label direction scores $\tfrac{1}{2}$ by construction, and the quantity of interest is exactly how much separation the estimator manufactures from the noise it was fitted to. Scoring in sample is what exposes it. Pooling the four models, the excess AUROC a mass-mean direction extracts from pure noise scales as
The exponent follows from the estimator alone. $\hat\theta$ is a difference of two sample means, each concentrating at the $\sqrt{N}$ rate, so the noise it carries has magnitude $O(\sqrt{d/N})$ whatever the activations look like, and the excess must fall as $N^{-1/2}$ at fixed $d$. Measured across four models and $N$ from $100$ to $32{,}000$, it comes back $-0.50$. The phenomenon is dimensional slack, the room that $d \gg N$ leaves for a direction built from a sample’s own fluctuations to separate that sample, since the fluctuations that define the direction are the ones being scored. It is not capacity.
Inverting the law gives the condition for noise alone to contribute less than $\varepsilon$ of excess AUROC,
which at the widths in this post and the standard 7B width gives the following. The table uses the unrounded fit, $a = 0.0456$ and $b = -0.497$, and the rounded values reproduce each entry to within about $2\%$.
| $d_{\text{model}}$ | $N$ for $\varepsilon = 0.05$ | $\varepsilon = 0.02$ | $\varepsilon = 0.01$ |
|---|---|---|---|
| $512$ | $849$ | $5{,}359$ | $21{,}597$ |
| $2{,}048$ | $3{,}396$ | $21{,}436$ | $86{,}390$ |
| $2{,}560$ | $4{,}245$ | $26{,}795$ | $107{,}987$ |
| $4{,}096$ | $6{,}792$ | $42{,}873$ | $172{,}780$ |
| $8{,}192$ | $13{,}584$ | $85{,}746$ | $345{,}560$ |
The curated true/false datasets in this literature hold $N \approx 1{,}200$ statements. For a $d = 4096$ model, the width of the 7B models these probes are usually run on, noise alone buys in-sample AUROC $0.55$ until $N \approx 6{,}800$ and $0.52$ until $N \approx 43{,}000$. Every dataset in the benchmark is between five and thirty times too small for an in-sample AUROC in the seventies to mean what it appears to mean. Held-out scoring removes the inflation but not its source. The fitted direction is the same object either way, and at these sizes it is mostly noise: the synthetic control in Empirical setup puts its overlap with the true direction at $0.24$ for the post’s own $d/N_{\text{train}}$. Held-out evaluation reports that attenuated direction’s honest score, which is why held-out $d'$ errs low where in-sample $d'$ errs high.
The control also behaves correctly at the other end of the ladder. On pythia-70m the true and shuffled curves lie on top of each other at every $N$, which is expected from a model with no recoverable direction.
The excess here falls approximately as $N^{-1/2}$ on every dataset tested, with fitted exponents between $-0.41$ and $-0.48$. For cities and larger_than, the excess $\pi^{-1/2}\sqrt{\mathrm{PR}/N}$ is set by the participation ratio of the within-class covariance rather than by $d$, and is predicted with no free parameter. counterfact also satisfies this prediction once the statements dropping massive activations are excluded from the dataset, while sp_en_trans sits $36$–$60\%$ above it. The derivation and the per-dataset fits are in the appendix The effective-dimension prefactor.
A causal test of the mass-mean direction
The decoding analysis says when a truth direction is readable. Steering asks whether it is causal, that is, whether displacing the residual stream
along $\hat\theta$ moves the model’s behavior toward the true completion. The two
need not agree. On counterfact_true_false at the deep layers of pythia-2.8b the
mass-mean direction is neither readable nor correctly causal. For the readout, the held-out AUROC at layer $28$ is $0.502$ against a null of
$0.560$, and it stays inside the null at every layer. As an intervention
it produces a significant effect, but with the incorrect sign.
As defined above, steering is reported as the antisymmetric response $A$ in units of the class-gap. A genuine truth direction produces $A > 0$, and the random-direction steering null fixes the bar $A$ must clear. Across ten seeds at layer $28$, at a displacement of one class gap, the mass-mean direction returns
where $p$ is the fraction of the $400$-draw random-direction null of $A(1)$ at least as extreme in the same direction, evaluated seed by seed.
The linear steering susceptibility $\chi$, the through-origin slope of $A$ against $h$ over $h \le 4$, is $-0.017$ at this layer. It is a diluted summary of the same effect. The response is not linear in $h$, and by $h = 4$, four class gaps out, $A$ has already fallen back toward zero, which is the regime The scale of an intervention warned about.
The slope of behavioral response against steering coefficient is the steerability of Tan et al. (2024), who fit this line through a logit-difference propensity curve. While their intervention displaces the last token position only, the harness here displaces every position, so magnitudes are not directly comparable. In this post, I additionally split the steering response into even and odd parts, to separate the degradation from the signed effect and compare against a random-direction null to determine which susceptibilities are distinguishable from chance.
The seed median sits two and a half null standard deviations below zero, and the two positive seeds sit inside the null. The effect is not null. It is significantly wrong-signed. Displacing an activation toward the true-class centroid, along the very vector the estimator returns for truth, makes the model measurably less likely to produce the true completion.
Because the AUROC of $\hat\theta$ is inside the decoding null, it is expected to do nothing under intervention, and a small negative point estimate on its own would be indistinguishable from noise. What makes this a result rather than a null is that the effect clears a $400$-draw random-direction null in the wrong direction, reproducibly across seeds and at two layers.
Read naively, this is a failure of the causal linear picture. Displacing along the direction fit to read truth does not raise the probability of the true completion and, in fact, lowers it. I use the rest of this post to argue that the naive reading is wrong: the wrong sign is a property of the estimator, not of the activation geometry. This means that the same activations, at the same layer, contain a direction along which displacement moves the model toward the true completion which the mass-mean estimator does not return. Instead, it returns $\hat v_1$, the axis of largest within-class variance, with which $\hat\theta$ shares a cosine of $0.994$ at this depth. Replacing the mass-mean estimate by the Fisher rule, downweighting the salient axis, or deleting the salient axis outright, inside the same linear class, recovers the correct sign. The mechanism responsible for the plain estimator producing a negative direction at layer $28$ is not settled here. The claim is about which part of the geometry the estimator is aligned with, not about how much truth the layer encodes. How well the corrected direction reads is a separate question, quantified in the next section by its held-out AUROC against the decoding null.
The rogue dimension
Definition. I call $\hat v_1$ a rogue dimension when it dominates the within-class covariance, that is, $\hat\lambda_1/\operatorname{tr}\hat C \approx 1$ or equivalently $\mathrm{PR} \approx 1$ and $\hat\lambda_1/\hat\lambda_2 \gg 1$. This condition is a statement about the noise geometry alone and is basis-free and independent of any probe. The estimator only becomes involved through the separate question of whether $\hat\theta$ has aligned with it.
This is deliberately not the same object as the massive activations of Sun et al.,
which are individual coordinates whose magnitude far exceeds the median, nor the
rogue dimensions of Timkey & van Schijndel, which are coordinates that dominate
cosine similarity. Those are properties of the mean and of a basis. This is a property
of the covariance and of no particular basis. The distinction is important here. At
counterfact layer $8$ the two coincide: $\hat v_1$ carries $86\%$ of its mass on a
single coordinate, and that coordinate’s mean activation is $1446\times$ the median
across coordinates, a massive activation by their criterion. By layer $28$ the
eigenvector has delocalized, $37\%$ on its largest coordinate and $90\%$ across five,
so the rogue dimension is no longer any one neuron. cities at layer $28$ still has
the massive activation, the same coordinate at $563\times$ the median, and has no rogue
dimension at all. Its leading eigenvector holds half a percent of its mass on any
coordinate. A massive activation is nearly constant across statements,
so it inflates the mean without inflating the within-class covariance. It produces a
rogue dimension only when the remaining variance is small enough for it to dominate.
The diagnosis is in the spectrum of the within-class covariance. I diagonalize
$\hat C = \sum_i \hat\lambda_i \hat v_i \hat v_i^{\top}$ with sample eigenvalues
$\hat\lambda_1 \ge \hat\lambda_2 \ge \dots$ and eigenvectors $\hat v_i$,
and measure where the mass-mean direction sits relative to its leading eigenvector.
On counterfact at pythia-2.8b, layer $28$:
Therefore, the estimator has not returned a truth direction, it has returned $\hat v_1$, the dominant axis of the within-class noise. On this dataset there is essentially no class-gap signal for $\hat\delta$ to align with, so the finite-sample $\hat\delta$ is dominated by its projection onto the highest-variance axis. The participation ratio $\mathrm{PR} = (\sum_i\hat\lambda_i)^2/\sum_i\hat\lambda_i^2$ characterizes the spectrum on a dimension-free scale: $\mathrm{PR} \to 1$ when one eigenvalue dominates the spectrum and $\mathrm{PR} \to d$ when the spectrum is flat, so it counts the directions the noise effectively occupies.
Read across depth, this shows that what determines recoverability is not whether a
dominant axis exists. At layer $8$, cities is also aligned with a nuisance direction, with
$\hat\lambda_1/\operatorname{tr}\hat C = 0.723$, $\lvert\cos(\hat\theta,\hat
v_1)\rvert = 0.914$ and $d'_{\mathrm{mm}} = 0.11$. However, this alignment dissipates by the middle of the network, and the separation along the estimator grows with the class gap while the spread stays of order one. On counterfact the alignment with the noise axis continues through layer $30$ and $d'_{\mathrm{mm}}$ only rises in the last two layers. However, unlike for cities, the class gap does not grow in these layers. It shrinks from $6.41$ to $1.99$ between layers $30$ and $31$. Due to the fall in the spread from $70.3$ to $9.5$, $d'_{\mathrm{mm}}$ increases overall.
Contrasting the spreads and class gaps between the two datasets for two representative layers makes the mechanism fully clear. At layer
$8$ the counterfact class gap is $47$ times the cities gap but its separation $d'_{\mathrm{mm}}$ is still lower than cities, because its spread is $59$ times larger. By layer $28$
the counterfact gap is down to $2.4$ times the cities gap, but the spread is still $83$ times
larger, resulting in the separation being lower by a factor of thirty-five.
The rogue dimension is not a property of counterfact alone:
companies_true_false and common_claim_true_false carry the same signature at layers
$24$ and $28$, $\mathrm{PR}$ within $0.05$ of $1$ and alignment above $0.8$,
and escape it by layer $31$ where their class gap has grown, as the appendix Results
on the full twelve-dataset benchmark records.
Recoverability is determined by the class gap becoming large relative to the within-class spread. The two datasets reach that by opposite routes. On cities the gap outgrows the noise. On counterfact the spread collapses.
The rank-one direction $\hat\theta_\perp$ tracks the whitened direction: on counterfact it sits at chance through layer $24$ and
reaches $0.568$ at layer $28$, and on cities it is already at $0.919$ by layer
$12$. Deleting the axis and downweighting it do the same work.
There is a sharper way to explain why the plain probe cannot leave the band on
counterfact. When $\delta$ lies along $\hat v_1$ and $\hat v_1$ carries nearly all
of $\hat C$, a random direction $u$ reads the class gap and the noise through the
same coordinate, $u^{\top}\delta \approx \lVert\delta\rVert u_1$ and
$u^{\top}\hat C u \approx \hat\lambda_1 u_1^{2}$, so
and $u_1$ cancels. For every random direction the projected gap and the projected
noise are carried by the same component, so their ratio is the one $\hat\theta$ itself
gives, and the null collapses onto the mass-mean value. The sweep shows it
to the third decimal. On counterfact the held-out $d'_{\mathrm{mm}}$ is $0.082$,
$0.081$, $0.081$, $0.080$ at layers $8$, $20$, $24$, $28$, and the median of
the random-direction null at the same layers is $0.083$, $0.081$, $0.079$, $0.078$.
The plain probe is not merely located inside the null distribution. Its $d'$ is indistinguishable from
that of a random direction, and the spread above that median, $p_{95}$ from $0.094$ to $0.139$, is what the two percent of variance off the axis contributes. That is what aligned with the axis means
operationally, and it is why more data would not lift $\hat\theta$ clear of the band
while the alignment holds: the null and the estimate move together.
$\hat v_1$ is also well estimated, which is what makes the alignment of $\hat\theta$ with it a feature of $\Sigma$ rather than an accident of the sample. That $\hat v_1$ is a population axis and not a finite-sample artifact rests on two direct checks. Identifying the eleven statements coordinate-first, by the size of that one coordinate and without forming the covariance, returns the same eleven at every layer, and a two-point variance formula built from those eleven predicts $97$–$99\%$ of $\hat\lambda_1$. A direction that can be derived by two disjoint routes is not a finite-sample accident. $\hat\theta$, by contrast, is a poor estimate of $\theta$, precisely because the class-gap signal is weak.
The obvious objection is that this is a fact about Pythia. However, the same observables on
OLMo-2-1B, a different architecture and
training corpus at a third of the parameters, reproduce the pattern. Both cities and counterfact have the same qualitative behavior as on Pythia for decoding. However, on OLMo cities also has a dominant within-class axis at every layer that the estimator does not align with. The numbers are reported in the appendix Replication on OLMo-2-1B.
The origin of the rogue dimension
On counterfact the leading axis of the
covariance is generated by the statements on which the massive activation drops.
A coordinate that is exactly constant across statements contributes nothing to $\hat C$, however large it is. A coordinate that takes a large value $a$ on a fraction $1-p$ of statements and a much smaller value $b$ on the remaining $p$ contributes the variance of a two-point distribution,
which for small $p$ and large $a-b$ can be enormous. At counterfact layer $8$,
four coordinates qualify as massive activations under the magnitude criterion of
Sun et al. On each, $1187$ of the
$1198$ statements sit at a near-constant value and eleven sit far below it. Coordinate $1793$ reads $1076.3 \pm 2.7$ on the majority and
$38.4 \pm 2.1$ on the eleven. With $p = 11/1198$ the formula predicts $9800$ against a measured
coordinate variance of $9788$. Summed over the four coordinates it gives $11{,}271$
against $\hat\lambda_1 = 11{,}413$, and $\hat v_1$ carries $98.6\%$ of its mass in
their span. In this case $\hat v_1$ carries no truth content. It is, to within a few
percent, the indicator of which statements failed to receive the massive activation.
The mass-mean estimator’s $0.994$ alignment with it is an alignment with an eleven-out-of-1198
membership function.
The eleven statements are identifiable coordinate-first, by a rule that never touches the covariance, the class means, or the labels, and it returns the same eleven at every layer. Refitting with them excluded suppresses the dominant eigenmode at layer $28$, taking $\hat\lambda_1/\hat\lambda_2$ from $535$ to $1.1$ and $\lvert\cos(\hat\theta,\hat v_1)\rvert$ from $0.994$ to $0.063$, and it leaves held-out $d'_{\mathrm F}$ unchanged at $0.385$. At that layer whitening leaves nothing further for deletion to recover. Nothing rescues the plain estimator, whose $d'_{\mathrm{mm}}$ stays inside its null under every removal regime tested. How the eleven are identified, what removing them does to $d'$ at each layer, and how that comparison depends on the Ledoit–Wolf intensity are given in the appendix Identification and removal of the eleven outlier statements.
The massive coordinates themselves are not dataset-specific. The same coordinates
qualify on cities and counterfact alike. What differs is whether any statement
drops these coordinates. On cities none do, so the coordinates stay constant, contribute nothing to
$\hat C$, and leave no rogue dimension behind. That is the earlier observation that
cities carries the same massive activation without the same pathology, now with a
mechanism rather than a coincidence. The incidence is specific to the dataset but the
mechanism is general. The per-layer incidence on both models, including pythia-1.4b, is
in the appendix Identification and removal of the eleven outlier statements.
The dropped activation mechanism also predicts where the rogue dimension should vanish. The number of massive coordinates grows with depth, from three at layer $3$ to nine
at layer $21$, is eight from layer $22$ through layer $29$, falls to five at layer $30$, and
from layer $31$ on there are none. The count is the same on
both datasets at every layer. The rogue dimension disappears at layer $31$ as
does most of the class gap. $\lVert\hat\delta\rVert$ falls by about seventy percent
across that boundary, which implies that most of the apparent class gap on
counterfact was from the eleven statements with missing massive activations in the earlier layers of the network. cities carries the same coordinates over the same span, but they are never dropped by any statement in the dataset. Hence cities never acquires a rogue dimension at any depth.
pythia-1.4b tests whether this is only a statement about the top of the network. Half of it is. The massive activation dies at 1.4b’s final layer too, so vanishing at the top of the network is a general property of the residual stream rather than anything about truth. The other half is not. At 1.4b there is no rogue dimension at any layer: from layer $3$ on the participation ratio runs between $14$ and $35$ and the alignment $\lvert\cos(\hat\theta,\hat v_1)\rvert$ between $0.07$ and $0.41$, nowhere near the pinned condition at 2.8b. Layers $17$ to $23$ carry a single massive coordinate that two to four statements drop, so the mechanism fires and produces nothing. The statements dropping massive activations are necessary for a rogue dimension and not sufficient, now shown at two model sizes.
This is also why counterfact decoding does not improve monotonically with scale. The
rogue dimension is absent at 1.4b and present at 2.8b, so the larger model is the one
where the mass-mean estimator is captured.
While $\hat\theta$ and $\hat v_1$ are aligned geometrically, it is unclear whether this is causal. The steering sweep includes a condition that displaces along $\hat v_1$ itself, the leading eigenvector of the within-class covariance. Because $\hat v_1$ is fit after both class means are removed, it is defined purely by within-class scatter and carries no information about which statements are true. At layer $28$ the antisymmetric response to $\hat v_1$ is $-0.033$ against $\hat\theta$’s $-0.028$. At layer $20$ the two agree to within $10^{-3}$ ($-0.0165$ against $-0.0159$). At every depth measured they track each other to within a few thousandths. They decode alike too, at held-out $\mathrm{AUROC} = 0.511$ for $\hat v_1$ against $0.514$ for $\hat\theta$ in the six-layer rogue-dimension sweep, both inside the null. The $0.994$ alignment is a geometric statement. The steering agreement is its behavioral form: a direction fit without any reference to the labels moves the model as much as $\hat\theta$ does and in the same direction. That establishes that truth content is not what carries the effect.
This reframes the anomaly. Steering along $\hat\theta \approx \hat v_1$ is not steering
along truth. It is displacing the activation along the dominant axis of the within-class
noise, which perturbs the forward pass in a way that happens to suppress the true completion.
A nuisance direction carries no information about the label, so nothing requires its
behavioral effect to come out positive. Nothing requires it to be reproducible either,
but here it is: the same negative sign appears in eight of ten seeds, at two counterfact
layers, and at mid-depth on cities. Whatever produces it is systematic rather than
arbitrary, and the label-blindness of $\hat v_1$ says only that truth is not what
produces it.
If this account is right, removing the contribution of $\hat v_1$ should recover a correctly signed causal effect, by correcting the estimator’s alignment with the rogue dimension rather than by enriching the function class.
Susceptibility and the score gradient
Before testing this prediction, it is worth asking what the steering number reports. The susceptibility is a linear functional of the direction pushed. Writing $g = \big\langle \nabla_x \ell(x)\big\rangle$ for the mean gradient of the behavioral score over the evaluation set, the odd part of the response gives
to leading order, the even terms having been removed by the antisymmetrization. So steering in the small-$h$ window reports the overlap of $w$ with the single vector $g$ along which the model’s true-versus-false margin moves. It does not test whether $w$ is a truth direction.
Every susceptibility quoted above is therefore a projection of one vector.
Decomposing in the plane of the next section, $\hat\theta = \cos\varphi\,\hat v_1 + \sin\varphi\,\hat e_2$, linearity gives
and at $\cos\varphi = 0.994$ the first term dominates unless $\chi(\hat e_2)$ is two orders of magnitude larger. One consequence holds before any measurement: a direction that merely disrupts the computation contributes to $S$, not to $A$, since $A$ is odd by construction. A wrong-signed $A$ therefore requires a signed channel onto the margin, $\hat v_1 \cdot g < 0$, which the estimator would inherit in proportion to $\cos\varphi$. Whether that channel exists is a question the gradient answers directly.
This also makes the assumption behind the correction explicit. Projecting out $\hat v_1$ recovers the sign only if $g$ has little weight along $\hat v_1$, i.e. the behaviorally causal direction and the salient axis are close to orthogonal in activation space. When they are not, $\chi(\hat v_1)$ mixes a truth term with a nuisance term inseparably, so deleting the axis would delete real causal signal along with the artifact.
Measured. One backward pass per example gives $\nabla_x \ell$, and $g$ follows by averaging over the same $250$ contrastive pairs the steering measurements use. Because the gradient is computed per pair, every overlap below carries a bootstrap interval over those pairs rather than resting on the point estimate. I quote $95\%$ intervals from $10{,}000$ bootstrap resamples under a fixed seed, and the scale to keep in mind is that a random direction gives $\lvert\cos\rvert \approx 1/\sqrt{d} = 0.020$ at $d = 2560$.
At layer $28$, $\cos(g,\hat v_1) = -0.008$ with $95\%$ confidence interval $[-0.044, +0.032]$. The overlap is consistent with zero. That has two consequences. Projecting $\hat v_1$ out removes essentially none of the behavioral channel, which is what licenses the correction. A channel consistent with zero cannot produce the wrong sign at first order, so $A(\hat v_1) = -0.033$ at this layer is not a linear effect. It is beyond first order in the displacement.
The overlap with $g$ also separates the two estimators. On counterfact the whitened
direction has a positive overlap at every layer from $20$ on, $+0.049$
$[+0.025, +0.064]$ at $L20$, $+0.060$ $[+0.030, +0.073]$ at $L24$ and $+0.064$
$[+0.031, +0.079]$ at $L28$, while the plain direction is consistent with zero at
$L24$ and $L28$. On cities it is the plain direction that has the positive overlap
at layer $28$, $+0.051$ $[+0.039, +0.060]$, while the whitened direction does not. In
both datasets the direction that steers correctly is the direction whose overlap with
$g$ is positive, and the overlap is measured without displacing an activation. The
sign and not the magnitude is what carries this: at counterfact layer $20$ both
directions have resolved overlaps, the whitened one positive and the plain one negative.
The rank-one correction gives the sharpest test of this, because
$\hat\theta_\perp$ is exactly $\hat e_2$, so its susceptibility is predicted by
$\cos(g, \hat e_2)$ alone, measured before anything is steered. On counterfact the
prediction has the right sign at all six layers of the rogue-dimension sweep, and where
the overlap is resolved the steering is significant and agrees with it. At layer $8$
the overlap is negative, $-0.062$ $[-0.076, -0.037]$, the prediction is
$c\,(\hat e_2\cdot g) = -0.16$, and $\hat\theta_\perp$ steers significantly
wrong-signed, $A(1) = -0.105$ over ten seeds ($p = 0.04$ against a $100$-draw
null). At layers $24$ and $28$ the overlap is positive, $+0.053$ and $+0.069$,
and $\hat\theta_\perp$ steers correctly, $+0.043$ and $+0.055$ against a predicted
$+0.050$ and $+0.055$. Removing $\hat v_1$ corrects the sign only where
what remains of the class gap overlaps $g$ positively.
cities also reproduces the mid-depth dip from the other side. Its plain overlap turns
significantly negative at layer $20$, $-0.038$ $[-0.044, -0.030]$, which is the same
window where the steering sweep finds $\chi = -0.0079$, an independent confirmation of
an anomaly that the steering measurement alone left uncertain.
There is a geometric reading of the two directions that makes the inversion of Marks &
Tegmark’s assignment in Separability, capacity and readout less paradoxical than it first appears. A
readout is a covector and a steering direction is a vector. To first order a
displacement couples to one covector, the score gradient $g$, so a direction steers
correctly when it overlaps $g$. Marks & Tegmark’s assignment is to steer along $\delta$
and read with $\Sigma^{-1}\delta$. This is right whenever $\delta$ overlaps $g$, which
is the case on cities. On counterfact, $\hat\delta$ lies along a nuisance axis with
no overlap with $g$. $\Sigma^{-1}$ removes that axis and restores the overlap.
This does not show that the model’s own readout is closer to $\Sigma^{-1}\delta$ than
to $\delta$ in general. On cities the plain direction carries the overlap but the
whitened one does not.
Two cautions on reading these magnitudes, the curvature at the large-$h$ end and how much of $g$ the plane captures, are given in the methods appendix.
A corrected linear estimator
Two corrections to the estimator, whitening and projecting out $\hat v_1$, make the same prediction, and neither adds capacity. Whitening replaces $\hat\theta \propto \hat\delta$ with the Fisher direction $\hat\theta_{\mathrm F} \propto \hat\Sigma^{-1}\hat\delta$, which downweights the mean-shift component along high-variance axes, $\hat v_1$ chief among them. The blunter control simply projects $\hat v_1$ out, $\hat\theta_\perp \propto (I - \hat v_1 \hat v_1^{\top})\hat\delta$, removing only the rogue dimension.
The two look like corrections of different rank, since the projection is exactly rank one while $\hat\Sigma^{-1}$ is not. However, in this regime they act inside the same plane. Letting $P = \mathrm{span}\{\hat v_1, \hat e_2\}$ with $\hat e_2 = \hat\delta_\perp / \lVert \hat\delta_\perp \rVert$ defined as the unit vector along the part of the class gap orthogonal to the rogue axis, $\delta_1$ and $\delta_\perp$ can be written as $\delta_1 = \hat v_1^{\top}\hat\delta$ and $\delta_\perp = \lVert\hat\delta_\perp\rVert$ respectively. $\hat\Sigma$ can be restricted to $P$ as $\mathrm{diag}(\lambda_1, \bar\lambda)$. For a unit direction in the plane $u(\varphi) = \cos\varphi\,\hat v_1 + \sin\varphi\,\hat e_2$, the separation it achieves is a Rayleigh quotient in the single angle $\varphi$:
The mass-mean estimator sits at $\tan\varphi_{\mathrm{mm}} = \delta_\perp/\delta_1$ and the Fisher direction has plane components $(\delta_1/\lambda_1,\ \delta_\perp/\bar\lambda)$, so the two are related by a single lever arm $\kappa \equiv \lambda_1/\bar\lambda$:
This shows that whitening is simply a rotation within $P$. The measured $\lvert\cos(\hat\theta,\hat v_1)\rvert = 0.994$ puts $\varphi_{\mathrm{mm}} = 6.3^{\circ}$. With $\kappa = \hat\lambda_1/\hat\lambda_2 = 535$ this gives $\varphi_{\mathrm F} = 89.0^{\circ}$. The whitened direction is orthogonal to $\hat v_1$ to within a degree, which is why $\hat\theta_{\mathrm F}$ and $\hat\theta_\perp$ produce the same sign flip: in this regime they are the same vector, and the full-rank correction reduces to the rank-one one.
The quotient also predicts the size of the gain. Writing $r = \tan\varphi_{\mathrm{mm}}$,
which at $\kappa = 535$ and $r = 0.11$ gives $7.3$, a gain of $2.7\times$ in $d'$. Measured on counterfact at layer $28$,
$d'_{\mathrm{mm}} = 0.080 \to d'_{\mathrm F} = 0.384$, a factor of $4.8$. The formula
predicts the right order and underestimates the measured gain. Inverting it for an
effective $\kappa$ is worked through in the methods appendix, under The
$\kappa_{\mathrm{eff}}$ inversion.
Two conditions make the reduction valid, and together they are the criterion for this
failure mode. The spectrum must have one dominant eigenmode, $\mathrm{PR} \approx 1$
with $\lambda_1/\lambda_2 \gg 1$, or there is no plane to reduce to. Additionally,
$\varphi_{\mathrm{mm}}$ must be small, or the estimator is not at the foot of the cliff
and there is nothing to correct. Both are read off the activations and the fitted direction, and together they diagnose whether the estimator has aligned with $\hat v_1$. They do not by themselves say which
way either direction will steer. That is set by the overlaps with the score gradient,
$\hat v_1\cdot g$ for the mass-mean direction and $\hat e_2\cdot g$ for the
correction, which are also measured without displacing an activation. At counterfact
layers $8$ through $16$ the spectrum meets the criterion more strongly than at layer
$28$, yet the corrections do not restore the sign there, because $\hat e_2\cdot g$ is
negative or unresolved. The diagnosis from the spectrum and the sign from the gradient
are both available before any steering is run. The diagnosis has been tested on one
dataset (counterfact) that satisfies the criterion, and one (cities) that does not,
and held in both. Two further main-tier sets,
companies_true_false and common_claim_true_false, satisfy the criterion at layers
$24$ and $28$ but their statements have no relation structure from which to build a pair of completions, so the
behavioral score of The scale of an intervention cannot be formed. This is discussed in
Appendix: results on the full twelve-dataset benchmark. cities at layer $28$ violates both conditions ($\mathrm{PR} = 29.7$,
$\lvert\cos\rvert = 0.159$). There the formula predicts a gain of $1.00\times$ at
$\kappa = 1.42$ and $r = 6.2$, since both directions already sit on the broad top of
the quotient, and the measured $1.22\times$ is a small gain from the rest of the
spectrum, outside the plane. The formula stops applying where the regime ends, and the
regime control below draws the same boundary.
For counterfact both the whitened and projected corrections flip the sign of steering. Steering along the whitened direction at layer $28$
gives
with $7/10$ seeds individually clearing the null at $5\%$. Here $\chi$ coincides with $A(1)$ at $+0.030$, because the whitened response is linear in $h$ over the whole sweep. The corrected steering behavior is about the same size as the wrong-signed steering behavior it replaces, two null standard deviations against two and a half, but it is monotone where the uncorrected steering was not. The rank-one correction $\hat\theta_\perp$, measured independently in the rogue-dimension sweep, agrees. At layer $28$ its ten seeds give a median $A(1) = +0.055$ ($4.5\,\sigma_{\text{null}}$), all ten positive and each at $p < 0.0025$ against the same $400$-draw null, so on its own it carries $A$ from significantly negative to significantly positive, and it raises held-out decoding AUROC at layer $28$ from $0.51$, the mass-mean value inside the null, to $0.57$.
Steering without a rogue dimension
Marks & Tegmark steer along the feature direction and report that difference-in-means
directions are the most causally implicated of the probes they compare. At the deep
counterfact layers that ordering reverses. The mass-mean direction, which their
framework nominates as the feature, steers the model away from the true completion,
and the whitened direction, which it demotes to a decision boundary, steers the model
toward it. The rogue-dimension diagnosis explains why. There $\hat\theta$ has collapsed
onto $\hat v_1$ and is not tracking a feature at all, so the $\Sigma^{-1}$ meant to
sharpen a readout is instead doing the work of recovering the direction.
The inversion is a property of a regime and not a refutation of Marks & Tegmark. This is demonstrated by cities as the control. There the two directions behave as mass-mean
probing intends. At pythia-2.8b layer $28$, the same model and depth at which
counterfact inverts, the plain direction steers correctly in $10/10$ seeds
($\chi = +0.040$) while the whitened direction is weak and inconsistent
($\chi = -0.004$, $4/10$). Read across the whole
depth sweep, the two datasets mirror each other. On cities, plain steering clears
its null at layer $28$. On counterfact, plain steering clears its null at two layers, $16$ and $28$, and is
wrong-signed at both. At every other layer of either dataset it sits inside the null.
At layer $28$ the two datasets share the model, the depth and the estimator, and
their significant effects point in opposite directions. When no single eigenmode
dominates the within-class covariance, the mean difference is the causal feature and
the inverse covariance only adds estimation noise, which is what Marks & Tegmark report, on datasets of exactly this kind. The inversion is confined to the regime where $\hat\delta$ has aligned with $\hat v_1$, corresponding to the presence of a rogue dimension in the dataset.
Discussion
A mass-mean truth direction can align with a causal truth direction, but it can also contain a large component along the most salient axis of the activations, with a signal-to-noise
analysis required to tell the two apart on any given benchmark. In this work I have focused on four consequences of this observation. I started by demonstrating that for the standard datasets in this literature, in-sample
separation is inflated by dimensional slack that scales as $N^{-1/2}$ requiring one to form a null distribution from scoring random vectors on held-out data in order to assess the significance of the probe score. I then showed that when the class gap is weak for a particular dataset or model layer, the mass-mean
estimator aligns with the leading within-class eigenvector. On counterfact that direction fails to decode truth and steers behavior with a significant wrong sign, two and a half null standard deviations deep at one class gap. However, I show that the same activations used to obtain the mass-mean direction contain a direction that steers correctly, and it can be reached without leaving the linear class, either by the Fisher direction or by projecting out the
leading eigenvector.
Three things should be noted before comparing to other reports. The corrected effect I find is modest. At one class gap the plain direction’s median response is $-0.031$, two and a half null standard deviations, and the whitened direction’s is $+0.028$, two. While the correction flips the sign of the effect, the corrected effect is modest so the correction restores a sign rather than supplying a large causal lever.
The held-out AUROC of the whitened counterfact direction at its selected layer $31$ reads $0.72$ against $0.98$ at layer $29$ for the whitened cities direction so this is a weak readout. The overall claim is a corrected sign and a confirmed mechanism, not a recovered truth direction of practical use. The incorrect steering behavior from the original mass-mean estimator could also imply that a richer, perhaps nonlinear, intervention is required. However, in my analysis I found that there are two corrections to the linear estimator for which this is not needed, projecting out the nuisance direction and whitening, which both enable decoding and correct the steering behavior.
Finally, the criterion that diagnoses this failure, a within-class spectrum with one dominant eigenmode and the mass-mean direction aligned to it, has one confirmed positive result and one confirmed negative result. It identifies the alignment but not the sign of a correction, which is set by the overlap of the corrected direction with the score gradient. Additionally, the steering measurements are on Pythia alone, since OLMo replicates the geometry and the decoding but was not steered.
Relation to other reports
Four recent reports also examine steering failures in fitted directions. Here I compare the rogue-dimension account against each.
Braun et al. (2025) study contrastive activation
addition across thirty-six behaviors and find two predictors of whether steering
works: the mean pairwise cosine similarity among the per-example activation
differences, which measures whether those differences share a direction, and the
separability of positive from negative activations along the difference-of-means
line. Their separability index is the
discriminability $d'$ of the projection onto that line, which is the
$d'_{\mathrm{mm}}$ of this post. While these two reports share a predictor, they differ in
what they find at low $d'_{\mathrm{mm}}$. Every dataset in Braun et al.’s study steers with a
net positive effect, with low separability showing up as per-example scatter around
that mean. In those cases they conclude that the behavior is not represented by a
coherent linear direction.
counterfact at layers $16$ and $28$, with $d'_{\mathrm{mm}} = 0.08$, sits at the
unsteerable end of their scale but steers with a negative mean that clears its null. This implies
that the behavior is linearly represented, since at layer $28$ the whitened direction steers it correctly. What has failed is the estimator. However, the correction does not hold at every depth. At layer $8$ whitening leaves the effect inside the random-direction null, and projecting out $\hat v_1$ does not restore the sign but steers significantly wrong-signed. This is not an absence of linear signal. It is the sign of
$\hat e_2\cdot g$, which at that depth is negative, as shown in Susceptibility and the score gradient. The per-sample form of the anomaly is documented by
Tan et al. (2024), who find that for several
concepts, close to half the inputs steer in the direction opposite to the one intended.
The counterfact effect at layers $16$ and $28$ is that variance surfacing as a
wrong-signed dataset-level mean that clears its null, with a proposed mechanism.
Ying et al. (2026) compare two kinds of truth
directions: domain-specific directions, and a single
domain-general direction recovered by concept erasure across many truth types. They find steering
along a domain-specific direction improves truthfulness on held-out factual questions but
steering along the domain-general direction consistently degrades performance. Their account is one
of mixture: a direction trained to span many domains conflates factual variance with
sycophancy-related variance, so intervening along it moves several things at once.
That mechanism is not available here, since the estimator showing the steering failure is fitted on the
counterfact domain alone and with no second domain to mix in. The result in their study and the result in this post therefore agree that a direction fitted to truth can exhibit incorrect behavior during steering but disagree on the mechanism and on the remedy.
Torop, Masoomi & Dy (2026) find inverted steering
vectors in the attention-head outputs of Gemma 3 12B, Qwen 2.5 14B and Olmo 3 7B:
mean-difference directions that discriminate the concept well, $\mathrm{AUC} \ge 0.85$
by their candidate cutoff, and whose positive steering consistently suppresses it
across inputs. They separate this from Tan’s per-input anti-steerable examples and
from the low-discriminability failures Braun et al. describe. The steering inversion they observe is along a direction that decodes well while the counterfact steering inversion I observe is shown to steer along a nuisance direction that fails to decode, since the estimator $\hat\theta$ sits inside its decoding null. The mechanism found in this post, alignment with a label-blind variance axis, is unavailable when
the direction is discriminative. The two are the same failure of sign at opposite ends
of decodability, and the diagnostics differ accordingly. They diagnose it through a downstream inner-product response, while this post reads the spectrum of the within-class covariance. Whether the
rogue-dimension criterion says anything about their cases is open.
Liu (2026) finds an overthinking failure in medical QA that is linearly decodable at $71.6\%$ balanced accuracy. Liu subsequently finds that five families of fixed residual-stream linear steering, across twenty-nine configurations, all move the behavior by nothing measurable. This is the decode-without-cause shape arrived at from the other side, and Liu attributes it to representational entanglement. Two things separate it from the case here. The effect described by Liu is null rather than wrong-signed, so there is no sign to restore. The entanglement is inferred from the failure, whereas the nuisance direction responsible here is named in advance from the within-class spectrum and then removed. A null steering result cannot distinguish an entangled direction from one that is absent, while a significantly wrong-signed result with an identified direction can.
This post has treated steering as a diagnostic. The steering results themselves are analyzed in a second, forthcoming post, Steering Vectors and the Limits of Linear Response. There the object of interest is the vector $g$ itself. Starting from the identity $\chi(w) = c\,(w\cdot g)$, I ask how much of $g$ lies along the high-variance directions of the within-class covariance, what bound it puts on any linear steering direction, how much of that bound the best direction actually reaches, and how much of $g$ published steering vectors capture.
Appendix: AUROC and its relation to $d'$
The main text defines $\mathrm{AUROC}(u) = \Pr[z_1 > z_0]$ as the probability that a random true statement outscores a random false one. This is the quantity the random-direction null resamples. The name refers to a different construction, the area under the receiver-operating-characteristic curve. This appendix builds that curve, shows its area equals the scoring probability, and records the Gaussian special case behind $\mathrm{AUROC} = \Phi(d'/\sqrt{2})$. The same Gaussian link, together with a closed form for the Mahalanobis cosine as a function of $d'$, is derived by Ying, Hase & Kriegeskorte (2026), whose signal-to-noise ratio is $d'(u)$ under the pooled within-class covariance.
![AUROC two ways. Left: the class-conditional score densities $$p_0, p_1$$ with a threshold $$t$$ cutting each into its false-positive and true-positive tail; the mean gap in noise units is $$d'$$. Right: sweeping $$t$$ traces the ROC curve, whose shaded area equals $$\Pr[z_1 > z_0]$$; the dot is the ROC point for the threshold at left, and the dashed diagonal is chance.](/assets/figures/auroc_two_definitions.png)
The receiver operating characteristic is an inheritance from WWII radar and quantified how well a receiver’s operator could separate signal (an aircraft echo) from noise as the detection threshold was varied. The same curve was adopted wholesale by signal detection theory and then by psychophysics and statistics, which is why a truth-probe analysis and a 1940s radar set share an acronym.
Fixing a direction $u$ and reading the scalar $z = u^{\top}x$, a classifier is obtained by thresholding, $\hat y = \mathbb{1}[z > t]$, and sweeping the threshold $t$ from $+\infty$ to $-\infty$ traces out two functions of $t$:
Plotting the point $(\mathrm{FPR}(t), \mathrm{TPR}(t))$ as $t$ sweeps traces out a curve from $(0,0)$ to $(1,1)$. A direction that separates the classes lifts the curve above the diagonal: at every false-positive rate the true-positive rate is higher, so the curve passes near the top-left corner. A useless direction gives the diagonal $\mathrm{TPR} = \mathrm{FPR}$ itself. The AUROC is the area under this curve. It is $1$ for perfect separation, $\tfrac{1}{2}$ for the diagonal, and $\mathrm{AUROC}(-u) = 1 - \mathrm{AUROC}(u)$ since changing the sign of $u$ swaps the conditional distributions.
The geometric area and the probabilistic $\Pr[z_1 > z_0]$ are the same number, which the main text leans on. Writing the class-conditional densities as $p_1(z) = p(z\mid y=1)$ and $p_0(z) = p(z\mid y=0)$, with CDFs $F_1, F_0$, one can parametrize the curve by $t$: the height is $\mathrm{TPR}(t) = 1 - F_1(t)$ and the horizontal coordinate is $\mathrm{FPR}(t) = 1 - F_0(t)$, so $\mathrm{d}(\mathrm{FPR}) = -p_0(t)\,\mathrm{d}t$. The area, integrated as $\mathrm{FPR}$ runs $0 \to 1$ (i.e. $t$ runs $+\infty \to -\infty$), is
Reading $1 - F_1(t) = \Pr[z_1 > t]$ for an independent draw $z_1 \sim p_1$, the integral averages this over $t \sim p_0$:
the probability that a random positive outscores a random negative — the Wilcoxon– Mann–Whitney identity. Ties contribute a boundary term of measure zero for continuous $z$, and half a count apiece if one insists on discrete scores. This is also why AUROC is calibration-free: it depends only on the ordering of scores, not their scale or location, so any strictly monotone reparametrization of $z$ leaves it unchanged. Unlike accuracy, it needs no choice of threshold.
When the class-conditionals of $z$ are Gaussian and the classes balanced, $z_y \sim \mathcal{N}(u^{\top}\mu_y,\; u^{\top}\Sigma u)$, the difference $z_1 - z_0 \sim \mathcal{N}(u^{\top}\delta,\; 2\,u^{\top}\Sigma u)$, so $\Pr[z_1 > z_0] = \Phi\big(u^{\top}\delta/\sqrt{2\,u^{\top}\Sigma u}\big) = \Phi(d'/\sqrt2)$, recovering the identity quoted in the definitions. Away from that case the relationship between $d'$ and AUROC is only approximate, which is why the main text reports both rather than deriving one from the other.
Appendix: results on the full twelve-dataset benchmark
The main text carries cities and counterfact through the steering and
rogue-dimension analysis and the nine main-tier sets through the transfer matrix. The
sweep covers all twelve. At pythia-2.8b, each at its own selected layer:
| dataset | $L$ | plain AUROC | $d'_{\mathrm{mm}}$ | whitened AUROC | null $p_{95}$ |
|---|---|---|---|---|---|
neg_cities |
25 | 0.978 | 3.03 | 0.981 | 0.683 |
sp_en_trans |
27 | 0.980 | 2.68 | 0.974 | 0.673 |
neg_sp_en_trans |
28 | 0.974 | 2.83 | 0.994 | 0.677 |
cities |
29 | 0.973 | 2.93 | 0.983 | 0.721 |
smaller_than |
31 | 0.974 | 2.79 | 1.000 | 0.786 |
larger_than |
28 | 0.929 | 2.05 | 1.000 | 0.735 |
cities_cities_disj |
29 | 0.836 | 1.42 | 0.836 | 0.586 |
cities_cities_conj |
32 | 0.820 | 1.28 | 0.932 | 0.606 |
common_claim_true_false |
31 | 0.757 | 0.64 | 0.722 | 0.611 |
companies_true_false |
31 | 0.699 | 0.17 | 0.855 | 0.585 |
counterfact_true_false |
31 | 0.550 | 0.12 | 0.716 | 0.552 |
likely |
12 | 0.890 | 1.75 | 0.929 | 0.593 |
The two translation sets sit at $N = 354$ rather than $1198$ and clear their random null by
as wide a margin as the single-frame sets. Every templated set with a single frame clears its null comfortably. The two free-form sets are the bottom of
the table: common_claim_true_false at $d'_{\mathrm{mm}} = 0.64$ and counterfact_true_false at
$0.12$. companies_true_false is interesting because it has a plain $d'_{\mathrm{mm}}$ of $0.17$, a bit above counterfact’s floor, but a whitened AUROC of $0.855$, the largest gap between the two estimators in the table after counterfact. Together with its rising superposition curve, this
makes it a second instance of the rogue-dimension pattern in decoding. The spectrum confirms
it. At layers $24$ and $28$, companies_true_false has $\mathrm{PR} = 1.01$ and
$1.02$, $\hat\lambda_1/\hat\lambda_2$ near $2000$ and $1250$, and
$\lvert\cos(\hat\theta,\hat v_1)\rvert = 0.98$ and $0.93$, with in-sample $d'_{\mathrm{mm}}$ of $0.02$ and $0.03$. That is counterfact’s signature at the same depths. By layer
$31$, where selection lands, it has escaped: $\mathrm{PR} = 3.2$, the alignment down
to $0.18$, in-sample $d'_{\mathrm{mm}}$ up to $0.51$. Since the dataset includes only statements and labels, with no contrastive completions from which to build a behavioral score, it does not include a causal measurement. common_claim_true_false is a third
instance, locked at layers $24$ and $28$ ($\mathrm{PR} = 1.02$ and $1.05$,
alignment $0.95$ and $0.83$) and escaping by layer $31$ ($\mathrm{PR} = 12.1$,
alignment $0.21$). cities_cities_conj, by contrast, never locks ($\mathrm{PR}$ from
$20$ to $42$ across the same layers, alignment below $0.11$). So within Pythia the
rogue dimension is a property of at least three of the nine main-tier sets at
mid-depth. What separates the other two from counterfact is that their class gap
grows enough to leave the axis by the later layers of the network.
One caution about the companies_true_false row is that $d'$ of $0.17$ maps to $\mathrm{AUROC} \approx 0.55$
under the Gaussian identity, not the $0.699$ measured. This is the largest
departure in the table, and a sign that the projected class-conditionals on this
dataset are far from the Gaussian idealization.
Appendix: the effective-dimension prefactor
The dataset enters in the determination of the prefactor which quantifies the number of directions effectively occupied by the noise. A spectrum concentrated on a few axes leaves a random labeling less room to find a separator than a flat one does, so the ambient width $d$ is the right count only when the spectrum is flat. In general the count is the participation ratio of the within-class covariance, and the amplitude should collapse across datasets once $N$ is measured in units of it.
A second sweep tests the exponent and the prefactor across datasets. The control is scored in sample, so no half has to be held back and the grid can run to the full set. Repeating it on pythia-2.8b at each dataset’s own best layer, over a seven-point geometric grid from $N = 100$ to that dataset’s total (capped at $3000$) with sixteen label permutations per point, and fitting $\mathrm{AUROC}_{\text{shuffled}} - \tfrac{1}{2} \approx a\,(N/2d)^{b}$ on each, gives
| dataset | $N_{\max}$ | $a$ | $b$ | $R^{2}$ | $\mathrm{PR}$ | $\hat\lambda_1/\operatorname{tr}\hat C$ |
|---|---|---|---|---|---|---|
counterfact_true_false |
$3000$ | $0.056$ | $-0.468 \pm 0.019$ | $0.992$ | $8.5$ | $0.33$ |
counterfact_true_false, without the 31 statements dropping massive activations |
$2969$ | $0.065$ | $-0.449 \pm 0.015$ | $0.994$ | $53.7$ | $0.070$ |
cities |
$1496$ | $0.053$ | $-0.446 \pm 0.009$ | $0.998$ | $34.6$ | $0.11$ |
larger_than |
$1980$ | $0.027$ | $-0.484 \pm 0.014$ | $0.996$ | $8.9$ | $0.28$ |
sp_en_trans |
$354$ | $0.064$ | $-0.413 \pm 0.044$ | $0.947$ | $16.8$ | $0.23$ |
The exponents look scattered, and taken at face value cities sits six standard errors
from $-\tfrac12$. They are scattered not by dataset but by how far up in $N$ each
grid reaches, because the power law is asymptotic and its small-$N$ end is shallower
than $-\tfrac12$. Refitting counterfact over truncated windows makes this explicit:
Each of the other three then matches counterfact restricted to its own range of $N$.
cities stops at $1496$ and gives $-0.446$ against $-0.439$, larger_than
reaches $1980$ and gives $-0.484$ against $-0.458$, and sp_en_trans stops at
$354$ and gives $-0.413$ against $-0.364$, within its own error. The four agree within error, so the dataset-independence of the exponent is measured from the estimator.
For the amplitude, $a$ is the wrong statistic to compare across datasets, since each $a$ is defined at its own fitted $b$. It is also not a free parameter. Under shuffled labels there is no signal, so within a class $x \sim \mathcal{N}(0, \Sigma)$, and the mass-mean direction is a signed sum of the samples,
so $\hat\theta \sim \mathcal{N}\!\left(0, \tfrac{4}{N}\Sigma\right)$ and
The in-sample separation it produces is the projected class gap, which is its own squared length, over the projected within-class spread,
which is where the participation ratio enters: $\mathrm{PR} = (\operatorname{tr}\Sigma)^{2}/\operatorname{tr}\Sigma^{2}$ is the count of directions the noise occupies, and it equals $d$ only for a flat spectrum. With $\mathrm{AUROC} = \Phi(d'/\sqrt{2})$ and $\Phi(x) \simeq \tfrac12 + x/\sqrt{2\pi}$ at small argument,
so the collapse constant is predicted rather than fitted:
Measured, $C$ is flat to about $\pm 10\%$ within a dataset over a thirtyfold range in
$N$, and runs from $0.49$ to $0.65$ across counterfact (with the statements dropping massive activations excluded), cities and
larger_than, against a factor $2.4$ in $a$ and $6.0$ in $\mathrm{PR}$. The
dependence on the spectrum runs the way the count predicts and against the intuitive
direction: larger_than is the most concentrated set, $\mathrm{PR} = 8.9$ with
$28\%$ of the within-class variance on one axis, and it carries the smallest amplitude
of the three.
When the collapse constant is predicted using the full dataset, counterfact misses the prediction due to the $31$ statements dropping massive activations. One can see that they carry essentially all of the variance along $\hat v_1$ and likewise the excess kurtosis of the shuffled-label projections. Excluding them puts it back on the parameter-free line with cities and larger_than.
sp_en_trans does not join the collapse. It sits at $C = 0.77$ to $0.90$, well above
$1/\sqrt{\pi}$, where the other three bracket it. The derivation factors into two
independent steps, and measuring them separately localizes the miss. The distributional
step holds: $\mathrm{AUROC}$ tracks $\Phi(d'_{\text{in}}/\sqrt{2})$ to within $3\%$
on all four datasets (counterfact with the statements dropping massive activations excluded), this one included, despite its projections carrying the largest
excess kurtosis of the four ($+0.8$ to $+1.2$ against $-0.7$ to $+0.2$ elsewhere).
The discrepancy is in the second-moment step: the measured $d'_{\text{in}}$ runs
$1.47$ to $1.62$ times $2\sqrt{\mathrm{PR}/N}$ here, against $1.03$ to $1.17$
on the other three.
Appendix: identification and removal of the eleven outlier statements
The eleven outlier statements are identifiable without reference to $\hat v_1$. The set is defined coordinate-first, without referring to the covariance, the class means, or the labels. A coordinate is massive if $\lvert\operatorname{med}_i x_{ij}\rvert$ exceeds both an absolute threshold and a large multiple of the median activation, and a statement is flagged if it deviates from the modal value on any massive coordinate. This criterion returns exactly the eleven statements of the projection-based set, at every layer measured, on the $1198$ statements used throughout. On the $3000$-statement sample of the effective-dimension appendix it flags $31$. Layer-invariance makes it a property of the input rather than of any particular representation.
Refitting on a training half with the eleven excluded and scoring on the full held-out half leaves the mass-mean direction essentially unchanged but moves the whitened direction substantially at early and middle depth:
| layer | $d'_{\mathrm F}$, all | $d'_{\mathrm F}$, eleven removed | null $p_{95}$ |
|---|---|---|---|
| $8$ | $0.010$ | $0.129$ | $0.094$ |
| $12$ | $0.015$ | $0.162$ | $0.092$ |
| $16$ | $0.031$ | $0.163$ | $0.095$ |
| $20$ | $0.005$ | $0.385$ | $0.097$ |
| $24$ | $0.063$ | $0.136$ | $0.132$ |
| $28$ | $0.384$ | $0.385$ | $0.128$ |
$d'_{\mathrm{mm}}$ stays inside its null under every regime. The layer-$28$ row is the informative one. Removing the eleven moves the geometry enormously there — $\hat\lambda_1/\hat\lambda_2$ falls from $535$ to $1.1$ and $\lvert\cos(\hat\theta,\hat v_1)\rvert$ from $0.994$ to $0.063$ while $d'_{\mathrm F}$ does not move at all. At that depth whitening has the same effect on $d'_{\mathrm F}$ as deleting the contaminated statements, but from layers $8$ through $20$ whitening alone leaves $d'_{\mathrm F}$ inside its null, while explicit removal lifts it above the null by factors of $1.4$–$4$. Layer $24$ sits between the two regimes.
The difference between the two regimes is controlled by the shrinkage intensity. Sweeping $\rho$ on the uncleaned data reproduces most of the effect of deleting the eleven statements: at layer $16$, $d'_{\mathrm F}$ runs from $0.031$ at the Ledoit–Wolf $\rho = 0.137$ to $0.530$ at $\rho = 10^{-4}$. What limits $\hat\Sigma^{-1}$ at these depths is therefore not the contamination itself but the regularization. At the Ledoit–Wolf intensity the estimate is pulled far enough toward the isotropic target that the inverse barely downweights the leading directions.
The massive coordinates are not dataset-specific. At pythia-2.8b the same
four coordinates qualify at layers $8$–$16$, the same seven at layer $20$ and the
same eight at layers $24$–$28$, on cities and counterfact alike. On cities all of the statements include the massive coordinates at all layers in either model. The coordinates therefore stay constant and contribute nothing to $\hat C$, so there is no rogue dimension.
Pythia-1.4b also contains massive coordinates which are dropped by some statements in counterfact but never in cities. Upon removing the three counterfact statements with dropped massive activations at layer $18$, $\hat\lambda_1/\hat\lambda_2$ drops from $1.5$ to $1.4$ so there is no dominant eigenmode to suppress.
I do not know what distinguishes the eleven statements on pythia-2.8b from the others. They span both labels, all end in a period, and their token lengths sit inside the bulk of the distribution.
Appendix: replication on OLMo-2-1B
To check that the rogue dimension is not a fact about Pythia, I ran the observables of
The rogue dimension on OLMo-2-1B, which has a
different architecture, a different training corpus and a third of the parameters. counterfact
stays locked to the leading axis at every depth,
$\lvert\cos(\hat\theta,\hat v_1)\rvert$ between $0.64$ and $0.94$ from layer $2$ to layer
$16$, with $d'_{\mathrm{mm}} \approx 0.1$. Its plain probe never clears its own null (held-out
$\mathrm{AUROC} = 0.529$ at its best layer, against a null 95th percentile of $0.547$),
while whitening lifts it to $0.647$. cities clears the null, as it does in Pythia: the alignment
falls to $\lvert\cos\rvert \approx 0.3$ and
the plain probe reaches $0.920$. The superposition probe agrees from the other side.
Stripping the leading components raises $d'_{\mathrm{mm}}$ on counterfact
($0.07 \to 0.31$).
Appendix: methods in detail
Script names refer to the public repository, jasteinberg/truth-directions-steering-repo, which holds the scripts below, the shared library they import, and the committed artifacts each number is read from.
Models and activations. Pythia 70m, 410m, 1.4b and 2.8b, and OLMo-2-1B for the
cross-family check, run in float16 on Apple silicon (MPS) through HuggingFace
transformers. The activation $x$ for a statement is the residual stream at the
final token, taken from every layer in one pass via output_hidden_states.
Layer $L$ means the output of block $L$, and layer $0$ is the embedding.
Environment. Python 3.11.15, torch 2.10.0 with the MPS backend,
transformers 4.57.6, numpy 1.26.4, scipy 1.17.1, scikit-learn 1.8.0, on
macOS 26.5.2 (arm64). The backend matters more than usual: MPS kernels have changed
numerical behavior between torch releases, so exact reproduction of third-decimal
figures should pin these versions.
Data. The nine main-tier sets are capped at $1199$ and class-balanced, giving
$N = 1198$, the two translation sets sit at $N = 354$ (their natural size), and
likely is carried as a distractor. Caps are
applied before splitting, so every dataset in a comparison contributes the same $N$.
Transfer. A direction is fit on the source’s training half at the source’s selected layer and scored on the target’s held-out half at the target’s selected layer, so the two ends of a transfer sit at different depths whenever the selected layers differ. The diagonal is scored the same way and is held-out.
Split and estimators. One class-stratified $50/50$ split at seed $0$
(split_indices), fit on the training half and scored on the held-out half unless a
quantity is explicitly marked in-sample. The three directions are
each normalized, and each sign-oriented so that its training-half AUROC is at least $\tfrac12$. This is necessary because $d'$ is sign-blind and AUROC is not. $\hat\Sigma$ is the within-class covariance with each class centered on its own mean, under Ledoit–Wolf shrinkage toward $(\operatorname{tr}\hat C/d)\,I$. The raw $\hat C$ is singular whenever $N_{\text{train}} < d$, which is most of this study.
Nulls. The decoding null draws $200$ random unit directions per layer
(Gaussian, then normalized) and scores them on the same held-out points as the
fitted direction, so the comparison is not confounded by the split. The
null-distribution figure above draws its own $400$ in-sample directions on the full
set, as an illustration of the distribution’s shape. The $200$-draw held-out null is
the one every result is scored against. The steering null
displaces along random unit directions exactly as the probe directions are displaced. Its draw
count is $400$ at counterfact layers $20$, $24$, and $28$ and cities layer
$28$, and $100$ at every other cell, including cities layers $12$ and $24$. The decoding null is reported as the $95$th percentile of the folded AUROC.
Steering clearances are signed: every steering $p$ in this post is a one-sided
rank of the direction’s mean $A(1)$ among the null draws in that direction, and
“clears the null” means $p < 0.05$ — the same test the figure bands denote as the
signed 5th–95th percentiles. Every claim of signal in this post is a claim about
that margin, never about the raw score.
Layer selection. The reported layer maximizes the margin $m$ of Separability, capacity and readout — held-out plain AUROC minus the layer’s own null $p_{95}$, not raw AUROC — so a layer with a large null cannot win by inflation. Layer $0$ is excluded: on templated statements the final-token embedding is nearly constant within a dataset, giving a degenerate $d'$. This margin criterion follows the variance-ratio selection of Bürger et al. and MacDiarmid et al. as adopted by Bao et al., with the null margin replacing the raw ratio.
Superposition probe. PCA of the activations, project out the top $k$ components for $k \in \{0,1,2,4,8,16,32,64\}$, refit the mass-mean direction in the residual subspace, and report $d'_{\mathrm{mm}}$ against $k$. The PCA basis is fit on the training half only.
Massive coordinates and the statements dropping them. Two conventions for “massive” appear in this post. In the rogue-dimension section a coordinate’s magnitude is quoted as $\lvert\text{mean}_i\, x_{ij}\rvert$ divided by the median of that quantity across coordinates — the $1446\times$ and $563\times$ figures. The test for statements dropping massive activations instead calls coordinate $j$ massive when $\lvert\operatorname{med}_i x_{ij}\rvert$ exceeds both $100$ in absolute terms and $100\times$ the median activation magnitude $\operatorname{med}_{ij}\lvert x_{ij}\rvert$, and flags statement $i$ as dropping the massive activation when $\lvert x_{ij} - \operatorname{med}_i x_{ij}\rvert > \tfrac12\lvert\operatorname{med}_i x_{ij}\rvert$ for some massive $j$. The two ratios agree to about one percent on this data ($1446$ against $1456$ at layer $8$ and $563$ against $560$ at layer $28$). The relative factor is $100\times$ rather than the $1000\times$ of Sun et al. because the dominant coordinate runs $1456$–$1693\times$ the median at layers $8$–$20$ but only $564$–$587\times$ at layers $24$–$28$, so the literal constant stops firing at exactly the depths under discussion. Every value between the two gives the same flagged set.
Shuffled-label control. The labels are permuted and the mass-mean direction refit.
This control is scored in-sample by design: a held-out shuffled direction scores
$\tfrac12$ by construction, which would hide exactly the inflation the control exists
to expose. The pooled fit $0.046\,(N/2d)^{-0.50}$ is run on counterfact across all
four models with one permutation per $(\text{model}, N)$ point, thirty-six points in
all, of which the three pythia-70m points whose in-sample excess came out negative are
dropped because the fit is in log space. The per-dataset fits are a separate sweep on pythia-2.8b alone, at each
dataset’s own best layer, over a seven-point geometric grid from $N = 100$ to that
dataset’s own total, with sixteen independent label permutations per $N$. Because the
control is in-sample there is no held-out half to reserve, so the grid runs to the
complete set rather than stopping at a shared cap. The within-class spectrum is recorded
both on the full set and on the first subsample at each $N$, so the participation ratio
can be checked for drift with sample size.
Steering. The intervention adds $h\,c\,w$ to the block output at layer $L$ at
every token position, for $h \in \{0.5, 1, 2, 4\}$ and both signs, with
$c = \lVert\hat\delta\rVert$ estimated on the training half, so $h = 1$ displaces
an activation by the distance between class means. The behavioral score is
$\ell = \log P(\text{true completion}) - \log P(\text{false completion})$, summed over
completion tokens given the prompt. Pairs are drawn as $250$ per seed from a fixed pool
of $400$. Ten seeds are used per cell, except the rogue-dimension sweep of $\hat v_1$ which has three for every layer except for layer $28$ which has $10$. $A$ and $S$ are the odd and even parts of the response in $h$, and
$\chi$ is the through-origin least-squares slope of $A$ against $h$ over the four
magnitudes, so $\chi$ is weighted toward the large-$h$ end and is not a pure
$h\to0$ derivative. Rank $p$ values are computed on $A$ at $h = 1$, each seed
against the null distribution of $A(1)$ over random directions, and not on $\chi$.
Ranking the seed-median $\chi$ against the null of $\chi$ gives $p = 0.018$ ($0.060$ for the seed mean) for the plain direction at
counterfact layer $28$, because the slope averages in the turned-back response at
$h = 4$.
Score gradient. $g = \langle \nabla_x \ell \rangle$ is computed by replacing the
block-$L$ output with a leaf tensor that requires grad, so backpropagation runs only
through layers above $L$, and by summing the gradient over token positions — the
quantity that matches an intervention applied at every position. A loss scale of
$1024$ prevents float16 underflow in the backward pass. The score $\ell$, the pairs,
the pool and the seed selection are taken from the steering harness unchanged, so the
two measurements refer to the same object. Per-pair gradients are retained, and every
overlap is reported with a $95\%$ bootstrap interval over pairs, $10{,}000$ resamples.
Scripts.
| claim | command | artifact |
|---|---|---|
| scale ladder, layer sweeps, transfer, superposition, Cover | recoverability.py sweep |
snr_sweep.json |
| OLMo decoding, nulls, superposition | recoverability.py sweep run on OLMo-2-1B |
olmo_goNogo.json |
| distractor transfer | recoverability.py transfer-likely |
transfer_likely.json |
unsteered judgment readout on cities, common_claim, counterfact |
recoverability.py verdicts |
check_model_verdicts.json |
| per-dataset shuffled-label fits and their spectra | dimensional_slack.py by-dataset |
cover_by_dataset.json |
| the shuffled-label law figure | figures/dimensional_slack.py shuffled-law |
truth_shuffled_law.png |
| the effective-dimension collapse figure | figures/dimensional_slack.py pr-collapse |
truth_pr_collapse.png |
shuffled-label sweep on counterfact without the statements dropping massive activations |
dimensional_slack.py no-droppers |
cover_no_droppers.json |
| excess kurtosis at layers $30$ to $32$, with and without the statements dropping massive activations | dimensional_slack.py kurtosis-droppers |
kurtosis_droppers.json |
| $d'_{\text{in}}$ against $2\sqrt{\mathrm{PR}/N}$, and AUROC against $\Phi(d'/\sqrt2)$ | dimensional_slack.py gaussian-check |
cover_gaussian_check.json |
| pool-size control on the collapse constant | dimensional_slack.py pool-control |
cover_pool_control.json |
| synthetic in-sample versus held-out $d'$ at planted $d' = 1$ | dimensional_slack.py insample-attenuation |
check_insample_attenuation.json |
| steering sweeps and $\chi$ | causal_steering.py sweep, causal_steering.py chi |
steer_ckpt/ |
| $400$- and $100$-draw steering nulls | causal_steering.py extend-null |
steer_ckpt/*__NULL.json |
| spectra, PR, alignment | rogue_dimension.py observables |
geometry_observables.json |
| the same observables at every layer | rogue_dimension.py all-layers |
geometry_all_layers.json |
| massive coordinates and the statements dropping them, at every layer | rogue_dimension.py massive |
massive_all_layers.json |
the same observables on companies, common_claim, conj |
rogue_dimension.py extra-datasets |
geometry_extra_datasets.json |
| massive coordinates, the statements dropping them, cleaned held-out $d'$ | rogue_dimension.py outliers |
outlier_check.json |
| statements dropping massive activations against statements with an internal period | rogue_dimension.py dropper-periods |
dropper_periods.json |
| projections of the statements dropping massive activations onto $\hat v_1$, layers $30$ to $32$ | rogue_dimension.py dropper-projection |
dropper_projection.npz |
| rogue-dimension steering and decoding of $\hat v_1$, $\hat\theta_\perp$ | rogue_dimension.py steer-arms |
rogue_dimension.json |
| $\hat\theta_\perp$ seeds $3$ to $9$, and the ten-seed table | rogue_dimension.py steer-arms --arms theta_perp --seeds 3,4,5,6,7,8,9, rogue_dimension.py theta-perp-table |
rogue_theta_perp_seeds3-9.json, theta_perp_table.json |
| OLMo cross-family geometry | rogue_dimension.py olmo |
geometry_olmo.json |
| score gradient and overlaps | rogue_dimension.py gradient |
score_gradient_*.json |
| fixed-seed bootstrap intervals on the overlaps | rogue_dimension.py gradient-ci |
score_gradient_*.json |
| shrinkage intensity and its decomposition | corrected_estimator.py intensity, corrected_estimator.py decomposition |
shrinkage_intensity.json, shrinkage_decomposition.json |
| $\rho$ sweep and cross-validated $\rho$ | corrected_estimator.py rho-sweep, corrected_estimator.py rho-cv |
shrinkage_sweep.json, shrinkage_cv.json |
| the twelve datasets | fetch_geometry_of_truth.py |
|
| figures, one file per section | figures/framework.py, figures/recoverability.py, figures/dimensional_slack.py, figures/causal_steering.py, figures/rogue_dimension.py (all draws a file’s figures) |
truth_*.png |
Reading the gradient decomposition. The identity $\chi(w) = c\,(w \cdot g)$
comes with two caveats. First, it is a small-$h$ statement,
while the measured $\chi$ is fit through the origin over
$h \in \{0.5, 1, 2, 4\}$. The two agree to about three percent on cities at layer
$28$ — $c\,(\hat\theta \cdot g) = +0.041$ against a measured $\chi = +0.040$ —
and disagree by an order of magnitude on counterfact, $-0.000$ against $-0.017$.
Where the class signal is tiny, the fitted susceptibility picks up curvature at the
large-$h$ end, so the decomposition should be read for signs and orderings
rather than magnitudes.
Second, the rotation plane of the main text, $\mathrm{span}\{\hat v_1, \hat e_2\}$, contains the estimators but not the causal direction. At the four cells where a rank test is quoted, only $3$–$7\%$ of $g$’s norm falls in the plane (the range over all twelve layer–dataset cells is $2$–$8\%$), against the $\sqrt{2/d} = 0.028$ a random 2-plane would capture. This is a factor of two above chance for three of the cells which is nonzero but not most of the causal direction. So the plane describes the effect of the estimator but not the geometry of the causal channel.
The $\kappa_{\mathrm{eff}}$ inversion. The gain formula of A corrected linear estimator predicts $2.7\times$ at $\kappa = 535$ against a measured
$4.8\times$ on counterfact at layer $28$. Inverting for $\kappa$ gives
$\kappa_{\mathrm{eff}} \approx 1.9\times10^{3}$, well above
$\hat\lambda_1/\hat\lambda_2$. This is consistent with the class gap carrying most of
its orthogonal weight below the second eigendirection, which the two-dimensional
reduction cannot resolve.
Known limitations. Every steering cell carries ten seeds except the three-seed
rogue-dimension sweep. The steering null
carries $400$ draws at the four cells listed under Nulls and $100$ elsewhere, so at a
$100$-draw cell the rank $p$ resolves only to $0.01$ and a clearance near
$p = 0.05$ rests on five draws. Marginal clearances there should be read with
that resolution in mind. Inference
is float16 throughout, which is adequate for the effect sizes here but not for the
third decimal. The geometry observables are computed on the full set rather than the
held-out half, since they describe the data rather than a probe’s performance. And a
single split at seed $0$ underlies the decoding numbers. The steering and gradient
results carry seed variation, the decoding ones do not.
The whitened numbers are a lower bound. Every $d'_{\mathrm F}$ in this post uses
Ledoit–Wolf’s shrinkage intensity to estimate $\Sigma$, which minimizes
$\mathbb{E}\lVert\hat\Sigma - \Sigma\rVert_F^2$. That is not the objective the post
reports: what matters here is $d'$ of the direction $\hat\Sigma^{-1}\hat\delta$, which
depends on the inverse and on one particular direction in it. The two diverge, and not
subtly. Sweeping $\rho$ on counterfact at pythia-2.8b, $d'_{\mathrm F}$ is larger at
some smaller $\rho$ than at $\rho_{\mathrm{LW}} \approx 0.137$ at every layer measured.
Choosing $\rho$ by five-fold cross-validation inside the training half and scoring once
on the held-out half gives $0.530$ against $0.031$ at layer $16$ and $0.441$
against $0.384$ at layer $28$, with cities improving more modestly
($3.68 \to 4.39$ at layer $28$). I have kept Ledoit–Wolf throughout rather than
switching estimators mid-study, so the whitened direction here understates what whitening can
do. This does not weaken any of the comparisons the post makes, since they all ask whether
whitening beats the plain estimator and clears the null. However, it does mean the cross-validated
selection is not itself reliable at every depth: at layers $8$ and $12$ it lands on
the opposite end of the grid and returns values inside the null, because five-fold on a
training half of about $600$ rows scores $d'$ on roughly $120$ points, which cannot
resolve the ridge. Fixing that properly, via repeated cross-validation, or an objective
smoother than $d'$, is left open.
Appendix: notation
Data and model.
| symbol | meaning |
|---|---|
| $L$ | layer index |
| $d$ | residual-stream width |
| $x \in \mathbb{R}^{d}$ | activation at the final token of a statement |
| $y \in \{0,1\}$ | truth label |
| $N$, $N_{\text{train}}$, $N_0$, $N_1$ | dataset size, training split, per-class counts |
| $\pi_0$, $\pi_1$, $\hat\pi_0$, $\hat\pi_1$ | class priors |
Population and sample geometry.
| symbol | meaning |
|---|---|
| $\mu_0$, $\mu_1$, $\hat\mu_0$, $\hat\mu_1$ | class-conditional means |
| $\delta = \mu_1 - \mu_0$ | class-mean gap |
| $\Sigma$ | within-class covariance (class-weighted) |
| $\hat C$ | sample within-class covariance, each class centered on its own mean |
| $\hat\Sigma$ | Ledoit–Wolf shrinkage estimate of $\Sigma$, with intensity $\rho$ |
| $\hat\lambda_i$, $\hat v_i$ | eigenvalues and eigenvectors of $\hat C$, ordered $\hat\lambda_1 \ge \hat\lambda_2 \ge \cdots$; $\hat\Sigma$ shares the eigenvectors, with eigenvalues $(1-\rho)\hat\lambda_i + \rho\operatorname{tr}\hat C/d$ |
| $\mathrm{PR}$ | participation ratio, $(\sum_i\hat\lambda_i)^2/\sum_i\hat\lambda_i^2$ |
| $k$ | number of leading principal components projected out (superposition probe) |
Directions. All are unit vectors.
| symbol | meaning |
|---|---|
| $u$ | a direction that is read, giving the scalar $z = u^{\top}x$ |
| $w$ | a direction that is steered along, added to the activation |
| $\hat\theta$ | mass-mean (difference-in-means) direction, $\hat\delta/\lVert\hat\delta\rVert$ |
| $\hat\theta_{\mathrm F}$ | Fisher direction, $\propto \hat\Sigma^{-1}\hat\delta$ |
| $\hat\theta_\perp$ | projection-out control, $\propto (I - \hat v_1\hat v_1^{\top})\hat\delta$ |
| $\hat e_2$ | unit vector along $\hat\delta_\perp$, the class gap orthogonal to $\hat v_1$ |
| $\varphi$, $\kappa$, $r$ | angle from $\hat v_1$ in the plane, lever arm $\lambda_1/\bar\lambda$, $r = \tan\varphi_{\mathrm{mm}}$ |
Decoding.
| symbol | meaning |
|---|---|
| $z = u^{\top}x$ | projected score |
| $u^{\top}\mu_0$, $u^{\top}\mu_1$, $u^{\top}\Sigma u$ | projected class means, projected within-class variance |
| $d'$ | separation in units of its own noise, sign-blind, a property of a direction |
| $d'_{\mathrm{mm}}$, $d'_{\mathrm F}$, $d'_{\perp}$ | $d'$ along $\hat\theta$, $\hat\theta_{\mathrm F}$, $\hat\theta_\perp$, held-out unless marked |
| $d'_{\mathrm M}$ | $\sqrt{\delta^{\top}\Sigma^{-1}\delta} = \max_u d'(u)$, the Mahalanobis separation |
| $\mathrm{AUROC}$ | $\Pr[z_1 > z_0]$, sign-aware, $\Phi(d'/\sqrt{2})$ for Gaussian classes, balanced |
| $C(N,d)$, $f(N,d)$ | Cover’s separable-dichotomy count and its fraction $C/2^{N}$ |
Steering.
| symbol | meaning |
|---|---|
| $h$, $c$ | steering coefficient and its unit, $c = \lVert\hat\delta\rVert$ |
| $\ell(x)$ | behavioral score, $\log P(\text{true}\mid x) - \log P(\text{false}\mid x)$ |
| $\Delta(h)$ | mean shift in $\ell$ under $x \mapsto x + h\,c\,w$ |
| $A$, $S$ | antisymmetric and symmetric parts of $\Delta$ |
| $g$ | mean gradient of the behavioral score, $\langle\nabla_x \ell\rangle$ |
| $\chi$ | steering susceptibility, $\mathrm{d}A/\mathrm{d}h$ at $h \to 0$, equals $c\,(w\cdot g)$; measured as the through-origin slope of $A$ against $h$ |
| seed / draw | resampling of the train/test split / of a random direction |
Corrected October 6, 2026: Due to a layer-indexing error, interventions and gradients were applied one layer after the layer the directions were fitted at, and the top layer was read after the final norm. The numbers throughout the steering, rogue-dimension and OLMo sections and the appendices, plus a few layer-specific statements have been corrected. However the qualitative conclusions are unchanged.
References
A fuller, annotated version of this bibliography is at the geometry of truth probes.
Separability, capacity and readout
- Cover, Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition — IEEE Trans. Electronic Computers EC-14(3):326–334 (1965) PDF.
- Diedrichsen, Berlot, Mur, Schütt, Shahbazi & Kriegeskorte, Comparing Representational Geometries Using Whitened Unbiased-Distance-Matrix Similarity — arXiv:2007.02789, Neurons, Behavior, Data Analysis, and Theory (2021).
Truth / honesty directions — reproduction targets
- Marks & Tegmark, The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets — arXiv:2310.06824, COLM 2024.
- Bürger, Hamprecht & Nadler, Truth is Universal: Robust Detection of Lies in LLMs — arXiv:2407.12831, NeurIPS 2024.
- Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (CCS) — arXiv:2212.03827, ICLR 2023.
The critiques — identifiability / which direction did you actually find (the SNR angle)
- Farquhar, Varma, Kenton, Gasteiger, Mikulik & Shah (DeepMind), Challenges with Unsupervised LLM Knowledge Discovery — arXiv:2312.10029 (2023).
- Roger, What Discovering Latent Knowledge Did and Did Not Find — AlignmentForum, 2023.
- Mallen & Belrose, Eliciting Latent Knowledge from Quirky Language Models — arXiv:2312.01037 (2023).
- Bao et al., Probing the Geometry of Truth: Consistency and Generalization — ACL Findings 2025.
- Ying, Ravfogel, Kriegeskorte & Hase, The Truthfulness Spectrum Hypothesis — arXiv:2602.20273 (2026).
- Ying, Hase & Kriegeskorte, Comparing Linear Probes with Mahalanobis Cosine Similarity — arXiv:2606.19603 (2026).
- Poulis, Crovella & Terzi, Testing the Limits of Truth Directions in LLMs — arXiv:2604.03754 (2026).
Controls and methodology
- Hewitt & Liang, Designing and Interpreting Probes with Control Tasks — arXiv:1909.03368, EMNLP 2019.
- MacDiarmid et al. (Anthropic), Simple Probes Can Catch Sleeper Agents — Anthropic Alignment Note, 2024.
Representation geometry — the rogue-dimension lineage
- Sun, Chen, Kolter & Liu, Massive Activations in Large Language Models — arXiv:2402.17762, COLM 2024.
- Timkey & van Schijndel, All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality — arXiv:2109.04404, EMNLP 2021 (pp. 4527–4546).
Models
- Biderman et al., Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling — arXiv:2304.01373, ICML 2023.
Steering directions
- Tan, Chanin, Lynch, Paige, Kanoulas, Garriga-Alonso & Kirk, Analysing the Generalisation and Reliability of Steering Vectors — arXiv:2407.12404, NeurIPS 2024.
- Braun, Eickhoff, Krueger, Bahrainian & Krasheninnikov, Understanding (Un)Reliability of Steering Vectors in Language Models — arXiv:2505.22637, ICLR 2025 Workshop on Foundation Models in the Wild.
- Torop, Masoomi & Dy, Inverted Detection and Control in Steering Vectors — arXiv:2608.02957 (2026).
- Liu, Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes — arXiv:2605.05715 (2026).
Claude (Anthropic) produced an initial draft from my description of the ideas, which I rewrote in full. It also supplied the numbers quoted in the text from the analysis artifacts, checked them, and implemented supplementary analyses we arrived at in discussion (the identification of the outlier statements, the shrinkage analysis, the extended nulls and the gradient intervals). I ran the main experiments and vouch for every claim here.