← All papers
First page of Centered Ablation: Why Last-Layer Directional Ablation Under-Fires or Collapses, and a Closed-Form Fix

Centered Ablation: Why Last-Layer Directional Ablation Under-Fires or Collapses, and a Closed-Form Fix

Quy-Anh Dang, Chris Ngo

cs.CL Nov 15, 2026 · v1
The proofs of the closed-form ablation results were formalized and machine-checked in Lean 4 with Mathlib, with assistant help.
Directional ablation removes a hidden state's component along a "behaviour direction" while a language model runs. It is the standard way to switch a behaviour such as refusal off, but it is erratic: on some models it barely fires, on others the text collapses into repetition. We show why, and fix it. At the last residual stream, ablation is exactly a logit bias: a fixed vocabulary footprint scaled by a gate, the cosine between the hidden state and the direction. The gate is measured from the origin, but last-layer states share a large common mode far from it, so a direction that overlaps the common mode fires on benign text and under-fires on its targets. Centered Ablation (CA) projects the common mode out of the direction, in closed form. Across 8 models, CA raises attack success by 29 and 46 points where the standard recipe under-fires, cuts collapse on Gemma-2-9B from 50% to 1%, and preserves 12 to 40 points more benign answers on high-overlap models, 39 to 46 more than the affine offset of prior work. Two numbers computed before generation predict, on models the analysis had not seen, which gain faithfulness and which gain attack success, and a collapse index marks when text degenerates. We also show that two standard safety judges score collapsed text in opposite directions, which can fabricate a 46-point effect.

Directional ablation, which removes a hidden state's component along a behaviour direction such as refusal, behaves erratically across language models. On some models it barely fires, and on others it causes text to collapse into repetition. Existing fixes do not explain these failures or predict them in advance.

At the last residual stream, ablation is shown to be exactly a logit bias: a fixed vocabulary footprint scaled by a gate, the cosine between the hidden state and the direction. Last-layer states share a large common mode far from the origin, so a direction that overlaps this mode misfires. Centered Ablation (CA) projects the benign common mode out of the difference-in-means direction in closed form. Diagnostics (overlap ω, η, and a collapse index β) are computed from cached activations, and the proofs are machine-checked in Lean 4 with Mathlib.

Across 8 models, CA raises attack success by 29 and 46 points where the standard recipe under-fires. It cuts collapse on Gemma-2-9B from 50% to 1%. It preserves 12 to 40 more benign answers on high-overlap models, and 39 to 46 more than the affine offset of prior work. The pre-generation diagnostics predicted the outcomes on models the analysis had not seen, and the paper shows that two standard safety judges score collapsed text in opposite directions.