← All papers
First page of Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization

Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization

Anna Mei, Michael R. Douglas

cs.LO Oct 14, 2026 · v1 cs.AI
Builds an agentic Lean autoformalization pipeline for the NIST DLMF and verifies Lean statements using LLM semantic review, numerical checks, and vacuity checks.
We introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness. Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-based semantic review with mechanical numerical and vacuity checks, providing checkable feedback and detecting errors missed by LLM judges. We apply this method, along with our Extract-Verify-Proof autoformalization pipeline, to construct the Formalized Library of Mathematical Functions (FLMF), a partial formalization of the NIST Digital Library of Mathematical Functions (DLMF). On three labeled datasets, our method achieves the lowest accepted-error rate among the evaluated methods, measured as the fraction of incorrect statements accepted by the verification method. Expert review of the results also identifies translation and source defects in existing datasets. Code available at https://github.com/annamei1/FLMF-Ch7ErrFunc-Release.git.

Agentic autoformalization can produce provable Lean statements that are unfaithful to the source. Examples include changed domains, signs, or branch conventions, and contradictory hypotheses. Existing faithfulness checks rely mostly on LLM or trained-model judges.

A layered verification stack combines an LLM semantic check (direct inspection plus back-translation of comment-stripped Lean), a mechanical numerical check of statements, and a post-proof vacuity check. This is integrated into an Extract–Verify–Prove pipeline with a dependency DAG and human gates. The pipeline is used to build FLMF, a partial Lean formalization of the NIST Digital Library of Mathematical Functions. The method is evaluated on 1,810 informal–formal pairs from FLMF, ProofNet, Beyond Compilation, and RLM25.

The method had the lowest accepted-error rate of the evaluated methods on all three labeled datasets, for example 4.8% on FLMF versus 15.7% for GPT-5.5. Expert review found translation and source defects in existing datasets and misprints in DLMF itself.

MethodFLMFBeyond CompilationProofNet
Ours5/105 (4.8%)3/154 (1.9%)14/153 (9.2%)
GPT-5.528/178 (15.7%)10/227 (4.4%)26/176 (14.8%)
Opus 4.672/232 (31.0%)24/261 (9.2%)44/206 (21.4%)
LeanScorer10/93 (10.8%)7/175 (4.0%)17/135 (12.6%)
Accepted-error rate (incorrect accepted / total accepted)