← All papers
First page of When Rank Rises as LLMs Degrade

When Rank Rises as LLMs Degrade

Zhaohui Geoffrey Wang

cs.LG Oct 7, 2026 · v1 cs.CL
One supporting theorem, the non-negativity of the anisotropy deficit via AM-GM on the spectrum, is machine-checked in Lean 4.24.0 with Mathlib and an axiom audit.
Post-training adapts language models in non-stationary environments. Practitioners monitor representation health with RankMe and related spectral statistics, often assuming that rank falls when representations degrade. We show that this assumption is unsafe for LLM post-training. In a controlled study of Qwen3-0.6B with four degradation modes and three seeds, data duplication worsens held-out loss by 75% relative to healthy while increasing both original and centred RankMe; the latter changes by 13.5 pooled standard deviations. Covariance effective rank rises to nearly twice its healthy value. This failure is spectral dispersion rather than collapse, so a one-sided monitor rates the worst checkpoint as the healthiest. By contrast, a learning-rate misconfiguration lowers centred RankMe and k95, while uncentred RankMe is inconsistent across seeds. Direction is therefore a property of the regime-statistic pair and cannot be fixed by recalibration alone. We also distinguish two often-conflated statistics: RankMe normalises singular values, whereas covariance effective rank normalises eigenvalues. On raw intermediate-layer states in the pretrained model, massive activations pin the latter near 1 out of dimension d while RankMe retains usable range. We then test a two-sided, multichannel sequential monitor with separate calibration and test data. In a pre-registered shared-prefix, leave-one-seed-out evaluation, it detects all three damage regimes in every fold 10 to 60 steps after the fork and separates dispersion from downward-rank damage by firing direction. However, it never precedes held-out probe loss, and calibration with two seeds produces false alarms on the held-out healthy seed. Spectral monitoring can diagnose failure regimes, but it does not warn earlier than held-out loss, and validity claims require held-out healthy data.

Practitioners monitor LLM post-training with spectral statistics such as RankMe, assuming rank falls when representations degrade. The work tests whether that assumption holds and whether such statistics can serve as a reliable label-free early-warning monitor.

A controlled study of Qwen3-0.6B uses four degradation modes and three seeds, tracking RankMe (uncentred and centred), covariance effective rank, k95, an anisotropy deficit and cross-batch similarity. The authors separate RankMe (singular values) from covariance effective rank (eigenvalues) and examine how massive activations affect each. A two-sided, multichannel sequential monitor with a union-bound false-alarm correction is evaluated under a pre-registered, shared-prefix, leave-one-seed-out protocol. Non-negativity of the anisotropy deficit is machine-checked in Lean 4 with Mathlib.

Under data duplication, held-out loss worsens by 75% while RankMe and effective rank rise (centred RankMe by 13.5 pooled s.d.); under learning-rate misconfiguration, the centred statistics fall. The monitor detects all damage regimes 10–60 steps after the fork and distinguishes dispersion from collapse, but it never precedes held-out probe loss and false-alarms on held-out healthy seeds.

modeprobe lossRankMe^cr_effk95
healthy2.530±0.01812.6±0.4174.8±1.6689.7±0.6
high_lr3.153±0.16799.9±3.5 ↓[3.6]183.2±4.2 [1.9, n.s.]665.7±5.9 ↓[4.1]
duplicate_data4.429±0.31877.9±4.8 ↑[13.5]329.2±14.4 ↑[10.7]782.0±7.0 ↑[13.1]
narrow_domain2.586±0.01816.4±0.8 ↑[4.2]179.4±1.1 [2.4]696.3±1.5 ↑[4.1]
Endpoint diagnostics after 1000 steps (mean ± s.d. over 3 seeds; separation from healthy in pooled s.d.)