← All papers
First page of Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture

Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture

Karen Sargsyan

cs.SE Nov 10, 2025 · v4 cs.AI
Lean 4 is used to formalize LORD online false-discovery-rate control, derive its error budget, and prove marginal and full FDR control.
AI-Scientist systems risk manufacturing spurious discoveries through uncontrolled multiple testing. We present a functional architecture that enforces statistical rigor at two levels: a Haskell embedded domain-specific language (the Research monad) that makes it impossible to test a hypothesis without updating the error budget, and a declarative scaffold that fixes the data flow and the statistical test, together with an OS-level sandbox that makes validation data physically absent from the environment in which LLM-generated code runs. We treat FDR control as a formal requirement and trace it to the implementation. We ground the design in a machine-checked Lean 4 formalization of LORD online false-discovery-rate (FDR) control: we derive its error budget and prove marginal FDR control, and full FDR control when thresholds do not adapt to earlier rejections. We then verify in SPARK/Ada that the LORD thresholds, computed in IEEE 754 arithmetic, never exceed the available wealth, given a margin condition that our configurations meet by a factor of at least eight; without a margin the property fails. To our knowledge this is the first machine-checked proof of an online FDR control theorem. In simulation, the architecture holds the false discovery rate near 1% against a 5% target, where a naive approach reaches 41%. In end-to-end case studies, a valid test avoids the false discoveries a flawed one produces, yet still finds real effects when the data allow. An adversarial evaluation confirms that, inside the sandbox, generated code cannot read the held-out data even when given its exact path.

AI-Scientist systems test many hypotheses automatically, which can produce spurious discoveries through uncontrolled multiple testing. Ad hoc imperative implementations of sequential FDR protocols are error-prone and can silently invalidate their statistical guarantees.

A Haskell eDSL (the Research monad) makes testing a hypothesis impossible without updating the error budget. A declarative scaffold fixes the data flow and the statistical test, and an OS-level sandbox keeps validation data physically absent from the environment where LLM-generated code runs. The LORD online FDR procedure is formalized in Lean 4, proving marginal FDR control in general and full FDR control when thresholds do not adapt to earlier rejections. SPARK/Ada is used to verify that IEEE 754 threshold computations never exceed the available wealth, given a margin condition.

The authors describe this as the first machine-checked proof of an online FDR control theorem. In simulation, empirical FDR stays near 1% against a 5% target, while a naive fixed-threshold approach reaches 41%. An adversarial test confirms that sandboxed code cannot read the held-out data, even when given its exact path.

ApproachTarget FDR/αEmpirical FDRPower
Naive (fixed α=0.05)0.050.40900.6399
Monadic (LORD++)0.050.01060.2900
Monte Carlo simulation (N=2000 hypotheses, 100 runs)