← All papers
First page of Sharp Bounds on the Winner's Bias under Independent and Nonnegatively Correlated Noise

Sharp Bounds on the Winner's Bias under Independent and Nonnegatively Correlated Noise

Ivan Viakhirev, Kirill Borodin, Grach Mkrtchian

cs.LG Oct 22, 2026 · v1 math.ST
All theorems, corollaries and lemmas on sharp winner's-bias constants are machine-checked in Lean 4 with Mathlib, in about 7,900 lines with no sorry.
Reporting the best of $M$ noisy, unbiased scores overstates the winner's value by up to $c\sqrt{V}$, where $V$ is the total noise variance and the sharp constant $c$ depends on how the noise is dependent. It was known only at the two extremes, arbitrary dependence ($\sqrt{(M-1)/M}$) and i.i.d. noise. We determine it for two classes in between: for independent, not identically distributed noise it is $\sqrt{(M-1)/(2M)}$, and for pairwise nonnegatively correlated noise it is $(M-1)/M$; for uncorrelated noise this answers the maximum version of an open question about the expected range. Explicit finite constructions attain both. As a corollary, the bias of the cross-validation estimator is bounded sharply within each class. All theorems are formally verified in Lean 4. Across 448 language-model and classifier selection pipelines, designs with per-candidate items realize a median 0.75 of the independent-class ceiling. At a fixed item budget, scoring all candidates on shared items is markedly less optimistic than giving each candidate its own items.

Reporting the best of M noisy, unbiased scores overstates the winner's value by up to c√V. The sharp constant c was known only for arbitrary dependence and for i.i.d. noise. The intermediate classes of independent non-identically distributed noise and pairwise nonnegatively correlated noise were open.

A centered master bound (Aven's bound applied to deviations from the mean, proved via Samuelson's inequality and Cauchy–Schwarz) isolates a covariance correction term. Explicit finite extremal constructions (a one-up law plus a common shock, and a cascade law) show the bounds are sharp. A transfer lemma carries the bounds to the cross-validation estimator. All results are formalized in Lean 4 with Mathlib (11 files, no axioms beyond the standard three), and an empirical study covers 448 language-model and classifier selection pipelines.

The sharp constant is √((M-1)/(2M)) for independent noise and (M-1)/M for pairwise nonnegatively correlated noise. This settles van Hasselt's conjecture on cross-validation bias and the uncorrelated case of Papadatos's question. Per-candidate evaluation designs realize a median 0.75 of the independent-class ceiling, and shared-item designs are markedly less optimistic at a fixed item budget.

MarbitraryCov≥0independenti.i.d. worst
20.7070.5000.5000.408
30.8160.6670.5770.516
100.9490.9000.6710.653
500.9900.9800.7000.696
Sharp constants c_M by dependence class