JPmHC Dynamical Isometry via Orthogonal Hyper-Connections
Biswa Sengupta, Jinhua Wang, Leo Brunswic
cs.LG
Feb 20, 2026 · v3
cs.AI
TL;DR
Lean 4 library certifies the free-probability Jacobian spectrum theory, Cayley iteration bounds, and algorithms, and also runs the Monte Carlo experiments as executables.
Abstract
The hyper-connection paradigm separates the skip-connections of transformers into $n$ parallel streams on which a linear mixer acts. Constraining the mixer to bistochastic matrices has proven effective in preventing gradient explosion but risk collapsing onto low-rank configurations, limiting expressivity. We therefore propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections). Our framework triages manifold constraints by computing the Jacobian singular spectrum of the end-to-end network in the mean-field limit and their spectral stability. It consists of: (i) an operator-valued free-probability library that predicts the Jacobian singular spectrum; (ii) an orthogonal mixer computed by a fixed-point Cayley iteration, which enables dynamical isometry; (iii) implicit differentiation of the Sinkhorn projection of the bistochastic baseline, with $O(1)$ backward operations; and (iv) a sensitivity theory giving the training budget before a bistochastic mixer collapses. The library is written in Lean, which certifies the mathematics and the algorithms. We validate JPmHC with TRM on ARC-AGI and nano-GPT on FineWeb-Edu: the theory predicts spectral collapse for TRM and stability for nano-GPT as observed in our experiments.
Problem
Hyper-connections split transformer skip-connections into parallel streams mixed by a learned matrix. Bistochastic mixers prevent gradient explosion, but their products can collapse to low rank, which limits expressivity and dynamical isometry.
Approach
JPmHC computes the end-to-end Jacobian singular spectrum in the mean-field limit using operator-valued free probability. It replaces bistochastic mixers with orthogonal ones computed by a fixed-point Cayley iteration. It also derives implicit differentiation of the Sinkhorn projection and a sensitivity theory giving the training budget before a bistochastic mixer collapses. The supporting library is written in Lean 4 and certifies the algebra, theorems and algorithms, including one proposition proved only in Lean.
Results
The nested theory matches Monte Carlo in all 12 mixer × depth cells tested. As the theory predicts, bistochastic mixers in TRM collapse, while nano-GPT's remain stable. On ARC-AGI the orthogonal mixer improves Pass@1 from 36.5% to 40.5%, and FineWeb-Edu validation loss is comparable.
| Metric | Bistochastic | Orthogonal |
|---|
| ARC-AGI Pass@1 | 36.5% | 40.5% |
| ARC-AGI Pass@10 | 46.8% | 53.1% |
| ARC-AGI Pass@1000 | 56.1% | 62.7% |
| FineWeb-Edu val. loss | 3.23 ± 0.008 | 3.24 ± 0.003 |
Bistochastic vs orthogonal mixers