Whose Effect Is It? How Objectives Inflate Learned Subspace Interventions
Toygar Tanyel, Serkan Ayvaz
cs.CL
Nov 3, 2026 · v1
TL;DR
The paper's first-order theoretical results on activation-patching effects are machine-checked in Lean 4 with Mathlib, with sources released on GitHub.
Abstract
Interpretability often locates concepts by copying part of a language model's state from one input into another (activation patching). A larger output change counts as stronger evidence that this part carries the concept. Methods that learn which part to copy often change the output more than the whole state does. By that reading, learned parts isolate the concept more cleanly. Yet the whole contains the part. So whose effect is this excess, the network's or the method's? Copying the whole state learns nothing, so it shows what the network produces. We prove that, to first order, the effects of orthogonal parts add up to that of the whole. A part therefore exceeds the whole exactly when the rest moves the output back, which does not require isolating the concept more cleanly. Objectives that reward a larger effect select such parts. Distributed alignment search with cross-entropy, for example, exceeds the whole-state copy just before the output on almost all tasks, and inside the network at most sites. In both, changing only its objective to target what the network produces removes most of the excess. It therefore comes largely from the objective, so a larger learned effect alone is not stronger evidence.
Problem
Learned subspace interventions such as distributed alignment search (DAS) often change a language model's output more than patching the whole state does. This raises the question of whether the excess comes from the network or from the training objective.
Approach
The authors prove that whole-state patching at a complete cut gives exactly the total effect, and that to first order the effects of orthogonal parts add up to the whole. A part can therefore exceed the whole only when the remaining part pushes the output back, and objectives that reward larger effects select such parts. These first-order statements are machine-checked in Lean 4 with Mathlib. They then evaluate DAS with four objectives on 90 tasks across 14 models, at the readout and at sites inside the network, and re-examine published benchmark results.
Results
With cross-entropy, DAS exceeds the whole-state effect in 1096 of 1124 eligible readout cases, at a median of 1.64–3.00 times the total effect per model. Per-pair squared-error objectives bring it to a median of 3% from the total effect, which shows that most of the excess comes from the objective. The authors propose a reporting check that compares each learned effect with whole-state patching at the same site.
| source | n | ce > T | ce / T | sq% |
|---|
| CausalGym | 404 | 404 | 2.79 | 2.4 |
| Kumon & Yanaka | 210 | 183 | 1.61 | 4.0 |
| RAVEL | 270 | 255 | 2.35 | 3.7 |
| all | 1168 | 1096 | 2.34 | 3.0 |
Readout results by source (cases, cases with margin above T, cross-entropy DAS above T, CE effect / T, squared-error distance from T in %)
