← All papers
First page of JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

JPmHC Dynamical Isometry via Orthogonal Hyper-Connections

Biswa Sengupta, Jinhua Wang, Leo Brunswic

cs.LG Feb 20, 2026 · v3 cs.AI
Lean 4 library certifies the free-probability Jacobian spectrum theory, Cayley iteration bounds, and algorithms, and also runs the Monte Carlo experiments as executables.
The hyper-connection paradigm separates the skip-connections of transformers into $n$ parallel streams on which a linear mixer acts. Constraining the mixer to bistochastic matrices has proven effective in preventing gradient explosion but risk collapsing onto low-rank configurations, limiting expressivity. We therefore propose JPmHC (Jacobian-spectrum Preserving manifold-constrained Hyper-Connections). Our framework triages manifold constraints by computing the Jacobian singular spectrum of the end-to-end network in the mean-field limit and their spectral stability. It consists of: (i) an operator-valued free-probability library that predicts the Jacobian singular spectrum; (ii) an orthogonal mixer computed by a fixed-point Cayley iteration, which enables dynamical isometry; (iii) implicit differentiation of the Sinkhorn projection of the bistochastic baseline, with $O(1)$ backward operations; and (iv) a sensitivity theory giving the training budget before a bistochastic mixer collapses. The library is written in Lean, which certifies the mathematics and the algorithms. We validate JPmHC with TRM on ARC-AGI and nano-GPT on FineWeb-Edu: the theory predicts spectral collapse for TRM and stability for nano-GPT as observed in our experiments.

Hyper-connections split transformer skip-connections into parallel streams mixed by a learned matrix. Bistochastic mixers prevent gradient explosion, but their products can collapse to low rank, which limits expressivity and dynamical isometry.

JPmHC computes the end-to-end Jacobian singular spectrum in the mean-field limit using operator-valued free probability. It replaces bistochastic mixers with orthogonal ones computed by a fixed-point Cayley iteration. It also derives implicit differentiation of the Sinkhorn projection and a sensitivity theory giving the training budget before a bistochastic mixer collapses. The supporting library is written in Lean 4 and certifies the algebra, theorems and algorithms, including one proposition proved only in Lean.

The nested theory matches Monte Carlo in all 12 mixer × depth cells tested. As the theory predicts, bistochastic mixers in TRM collapse, while nano-GPT's remain stable. On ARC-AGI the orthogonal mixer improves Pass@1 from 36.5% to 40.5%, and FineWeb-Edu validation loss is comparable.

MetricBistochasticOrthogonal
ARC-AGI Pass@136.5%40.5%
ARC-AGI Pass@1046.8%53.1%
ARC-AGI Pass@100056.1%62.7%
FineWeb-Edu val. loss3.23 ± 0.0083.24 ± 0.003
Bistochastic vs orthogonal mixers