← All papers
First page of Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference II: General convex penalties

Simultaneous analysis of approximate leave-one-out cross-validation and mean-field inference II: General convex penalties

Pierre C Bellec

math.ST Nov 16, 2026 · v2
The construction of the ALO matrix, its recovery identities, and the statistical ALO and mean-field bounds are formalized in Lean 4, with code in the arXiv ancillary files.
For smooth penalties and group Lasso, Part I (https://arxiv.org/abs/2501.02624) analyzed approximate leave-one-out (ALO) cross-validation by comparing its weights with the trace correction used in mean-field inference. For a $C^2$ penalty $R$, the matrix entering these corrections is $A=[\nabla^2R(\widehat b)+X^TDX]^{-1}$, where $\widehat b$ is the full-data estimator and $D$ is the diagonal matrix of fitted loss curvatures; for group Lasso, the penalty and loss curvatures are first restricted to the active support before taking the pseudoinverse. The obstacle to extending this argument to general convex penalties was the lack of an explicit formula for the matrix $A$ entering these corrections. For every finite-valued strongly convex penalty, we construct such a matrix from ordinary derivatives of the penalty response at the observed subgradient. Its projection–pseudoinverse formula separates the response space and penalty curvature, fixed by conditioning, from the explicit design dependence. This supplies the missing ingredient to extend the probabilistic argument of Part I. The construction recovers the inverse Hessian for smooth penalties and the active-group inverse for group Lasso, including inactive boundary groups. In the proportional regime, for Gaussian designs with general positive-definite covariance and covariance-scaled strong convexity, we obtain unconditional bounds comparing both ALO and mean-field predictions with exact leave-one-out predictions in regression and single-index models. The expected average squared prediction error is $O(n^{-1})$, and the expected absolute criterion error is $O(n^{-1/2})$, under bounded, 1-Lipschitz loss scores and suitable test functions. The matrix and ALO weights use only observed data; the mean-field correction additionally uses the design covariance. The proofs are formalized in Lean 4.

Part I analyzed approximate leave-one-out (ALO) cross-validation and mean-field inference for smooth penalties and group Lasso. Extending that analysis to general convex penalties was blocked by the lack of an explicit formula for the correction matrix A.

For finite-valued strongly convex penalties, the authors build a matrix A_der from ordinary derivatives of the penalty response (the gradient of the Fenchel conjugate) at the observed subgradient, written in a projection–pseudoinverse form. This form separates the response space and penalty curvature, which are fixed by conditioning, from the explicit design dependence. The conditioning argument of Part I is then reapplied. The recovery identities and the statistical guarantees are formalized in Lean 4.

A_der recovers the inverse Hessian for smooth penalties and the active-group inverse for group Lasso. For Gaussian designs in the proportional regime, both ALO and mean-field predictions are compared with exact leave-one-out predictions: the expected average squared prediction error is O(1/n), and the expected absolute criterion error is O(1/√n).