← All papers
First page of Removing spurious minima for planar features by skip connections

Removing spurious minima for planar features by skip connections

Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo, Georg Loho

cs.LG Oct 1, 2026 · v2 cs.AI math.OC
Main theorems on ReLU teacher-student loss landscapes are formally verified in Lean, with each proposition and theorem linked to a Lean artifact.
Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher–student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

It is not fully understood when overparameterization or skip connections remove spurious local minima in neural-network loss landscapes. The setting studied is the Gaussian population loss of shallow, bias-free ReLU networks in a teacher-student model.

Local minima are analyzed through a residual function built from closed-form Gaussian pair moments. The proofs reduce the skip model to a centered absolute-value model and use feature-learning (span confinement) and zero-mass arguments. For planar teachers they apply angular interlacing arguments on the circle. A computer-assisted strong-convexity certificate constructs spurious minima. The key results (Propositions 8, 9 and 12, Theorems 10, 11 and 13) are each tied to Lean formalizations.

With a learned linear skip, every local minimum with non-negative student output weights is an exact fit for positive, coplanar teachers once the student is at least as wide as the teacher. Without the skip, a 3-neuron teacher in dimension two has spurious minima at every student width of at least three. Critical points have at most 2m distinct student directions, and the benignity result transfers to empirical minima on parameter balls of fixed radius.