← All papers
First page of Beyond Correctness: Toward Automated Novelty Verification with Lean 4

Beyond Correctness: Toward Automated Novelty Verification with Lean 4

Ayrton Porto

cs.AI Aug 2, 2026 · v1
Autoformalizes LaTeX statements into Lean 4, searches Mathlib via Leandex, tests triviality with Lean tactics, and compares proof premise sets.
Artificial intelligence systems applied to mathematics verify correctness but not novelty: an automatically generated theorem can compile in Lean without errors and yet be an already known result. This article presents AViD Journal, a pipeline that receives a LaTeX article, formalizes its statements in Lean 4, and issues a novelty verdict through a decision tree over three dimensions: prior existence in a formal corpus (Mathlib) and an informal one (TheoremSearch and Matlas, with temporal filter and LLM judge), non-triviality via automatic tactics, and structural distance between proofs measured as Jaccard distance over premise sets. Evaluation on papers withdrawn from arXiv due to declared duplication produced a result more informative than any performance measure: the identification of three obstacles that limit the approach regardless of this implementation. First, successful compilation of a Lean file does not guarantee semantic fidelity. Second, the recall ceiling is imposed by the coverage of theorem indices, not by the similarity metric. Third, arXiv removes the source code of articles upon withdrawal, compromising the reproducibility of any benchmark built upon them.

AI systems for mathematics check whether a theorem is correct but not whether it is new. A generated theorem can compile in Lean and still be an already known result. Equating novelty with absence from Mathlib is a poor criterion: it flags trivial facts as new and misses results that exist only in the informal literature.

AViD Journal parses a LaTeX article, orders its theorem environments by their dependencies, and formalizes them in Lean 4 using one of several selectable LLM backends. A decision tree with eight possible verdicts then checks three dimensions. The first is prior existence in Mathlib (searched via Leandex) and in the informal TheoremSearch and Matlas indices, using a temporal filter and an LLM judge. The other two are non-triviality, tested by whether automatic tactics such as aesop, decide or norm_num close the goal, and proof distance, measured as Jaccard distance over premise sets. Each instrument is validated separately, and the whole pipeline is evaluated on arXiv papers withdrawn for declared duplication, each paired with matched controls.

The Jaccard distance was consistent with the human judge on 5/5 calibration pairs, and the triviality check succeeded on 22/24 cases. Backend choice mattered: Qwen 3.7-max formalized 5/5 test cases while both DeepSeek models formalized 0/5. The evaluation identified three obstacles that limit the approach regardless of implementation: successful compilation does not guarantee semantic fidelity, recall is capped by the coverage of theorem indices rather than the similarity metric, and arXiv removes the sources of withdrawn papers, undermining reproducibility of such benchmarks.

ModelSuccessPredominant failure mode
Qwen 3.7-max5/5—
GLM-5.23/5fixable errors, one timeout
DeepSeek V4 Pro0/5synthesis errors, placeholder definition
DeepSeek V4 Flash0/5synthesis errors, unexpected tokens
Formalization backend selection (5 test cases each)