← All papers
First page of Complex-valued Phase-Coherent Transformers

Complex-valued Phase-Coherent Transformers

Leona Hioki

cs.LG Sep 18, 2026 · v1
Global phase-rotation equivariance and depth-uniform stability of the attention family are formalised in Lean 4 with Mathlib, sorry-free, in the released repository.
Complex-valued Transformers have inherited softmax attention over the raw complex inner product. Outside natively complex domains this standard form stays near chance, and no complex attention had been shown to correct it. We show that the match must be a scaled cosine score: L2-normalise queries and keys, so the score reads their cosine similarity and ignores their magnitudes, and hold that score at order-one scale. With this the same models train on four diagnostic tasks under two different gates; without the normalisation they stay at chance on ListOps and Needle under both gates and fall far below on the other two, and a normalised score placed at too small a scale fails as well. The resulting family of phase-coherent Transformers (\PCT) matches or exceeds the strongest real-valued baseline across long-range memory, positional retrieval, hierarchical reasoning, frequency-domain classification and physical complex signals; it shows no degradation up to depth 20; and its loss decreases log-linearly over a 61-fold range of parameters. A member of the family, complex screening combined with a phase-coherent recurrence, is the first genuinely complex-valued neural network to solve Path-X, with 91.6% of its trainable parameters complex-valued against 38.2% for S4. We record these as signs of generalisation not previously seen in complex-valued neural networks.

Complex-valued Transformers that apply softmax to the raw complex inner product stay near chance on discrete tasks outside natively complex domains. Which property of the attention score path decides whether a complex Transformer trains had not been isolated.

Queries and keys are L2-normalised and scored by the real part of their complex cosine similarity scaled by sqrt(d_h), followed by a real-valued gate (sigmoid, L2-softmax, or screening). This defines the phase-coherent Transformer (PCT) family. The family is compared against parameter-matched real and complex baselines on nine benchmark rows. A complex screening gate is combined with an LRU-style phase-coherent recurrence for Path-X. Equivariance under global phase rotation and depth-uniform stability are machine-checked in Lean 4 with Mathlib.

PCT matches or exceeds the strongest real baseline (real screening) on most tasks, for example 0.854 vs 0.698 on ListOps. It shows no degradation up to depth 20, and its loss decreases log-linearly with scale. The screening plus recurrence model solves Path-X at 92.71% with 91.6% of its parameters complex-valued.

taskc_softmaxr_screenPCT
Copy d=20000.1001.0001.000
Needle L=20480.0001.0001.000
ListOps L=10240.1040.6980.854
LRA-Image (CIFAR)0.2320.3180.372
Unnormalised complex softmax vs PCT vs strongest real baseline (selected rows)