When Rank Rises as LLMs Degrade
Zhaohui Geoffrey Wang
cs.LG
Oct 7, 2026 · v1
cs.CL
TL;DR
One supporting theorem, the non-negativity of the anisotropy deficit via AM-GM on the spectrum, is machine-checked in Lean 4.24.0 with Mathlib and an axiom audit.
Abstract
Post-training adapts language models in non-stationary environments. Practitioners monitor representation health with RankMe and related spectral statistics, often assuming that rank falls when representations degrade. We show that this assumption is unsafe for LLM post-training. In a controlled study of Qwen3-0.6B with four degradation modes and three seeds, data duplication worsens held-out loss by 75% relative to healthy while increasing both original and centred RankMe; the latter changes by 13.5 pooled standard deviations. Covariance effective rank rises to nearly twice its healthy value. This failure is spectral dispersion rather than collapse, so a one-sided monitor rates the worst checkpoint as the healthiest. By contrast, a learning-rate misconfiguration lowers centred RankMe and k95, while uncentred RankMe is inconsistent across seeds. Direction is therefore a property of the regime-statistic pair and cannot be fixed by recalibration alone. We also distinguish two often-conflated statistics: RankMe normalises singular values, whereas covariance effective rank normalises eigenvalues. On raw intermediate-layer states in the pretrained model, massive activations pin the latter near 1 out of dimension d while RankMe retains usable range. We then test a two-sided, multichannel sequential monitor with separate calibration and test data. In a pre-registered shared-prefix, leave-one-seed-out evaluation, it detects all three damage regimes in every fold 10 to 60 steps after the fork and separates dispersion from downward-rank damage by firing direction. However, it never precedes held-out probe loss, and calibration with two seeds produces false alarms on the held-out healthy seed. Spectral monitoring can diagnose failure regimes, but it does not warn earlier than held-out loss, and validity claims require held-out healthy data.
Problem
Practitioners monitor LLM post-training with spectral statistics such as RankMe, assuming rank falls when representations degrade. The work tests whether that assumption holds and whether such statistics can serve as a reliable label-free early-warning monitor.
Approach
A controlled study of Qwen3-0.6B uses four degradation modes and three seeds, tracking RankMe (uncentred and centred), covariance effective rank, k95, an anisotropy deficit and cross-batch similarity. The authors separate RankMe (singular values) from covariance effective rank (eigenvalues) and examine how massive activations affect each. A two-sided, multichannel sequential monitor with a union-bound false-alarm correction is evaluated under a pre-registered, shared-prefix, leave-one-seed-out protocol. Non-negativity of the anisotropy deficit is machine-checked in Lean 4 with Mathlib.
Results
Under data duplication, held-out loss worsens by 75% while RankMe and effective rank rise (centred RankMe by 13.5 pooled s.d.); under learning-rate misconfiguration, the centred statistics fall. The monitor detects all damage regimes 10–60 steps after the fork and distinguishes dispersion from collapse, but it never precedes held-out probe loss and false-alarms on held-out healthy seeds.
| mode | probe loss | RankMe^c | r_eff | k95 |
|---|
| healthy | 2.530±0.01 | 812.6±0.4 | 174.8±1.6 | 689.7±0.6 |
| high_lr | 3.153±0.16 | 799.9±3.5 ↓[3.6] | 183.2±4.2 [1.9, n.s.] | 665.7±5.9 ↓[4.1] |
| duplicate_data | 4.429±0.31 | 877.9±4.8 ↑[13.5] | 329.2±14.4 ↑[10.7] | 782.0±7.0 ↑[13.1] |
| narrow_domain | 2.586±0.01 | 816.4±0.8 ↑[4.2] | 179.4±1.1 [2.4] | 696.3±1.5 ↑[4.1] |
Endpoint diagnostics after 1000 steps (mean ± s.d. over 3 seeds; separation from healthy in pooled s.d.)