What Has Actually Been Proved? Auditable Paper-to-Lean Alignment with LeanTrail
Fabricio Venturim, Leighton Estrada, Claudio Silva, Jorge Poco
cs.LO
Oct 14, 2026 · v1
cs.PL
TL;DR
Builds a tool that aligns paper items with Lean repository declarations and Blueprints, auditing transitive sorry dependencies to assign coverage states.
Abstract
Compiler feedback certifies that Lean accepts a declaration, not that the declaration captures the intended result: a repository can compile while a central theorem is weakened, merely auxiliary, or transitively dependent on sorry. Formalization progress should instead be measured against the source paper, in terms of its paper items: definitions, lemmas, and theorems. LeanTrail implements this view as a post-hoc, tool-agnostic audit: deterministic retrieval proposes candidate counterparts for each paper item in the Lean repository and, when available, the Blueprint; a language model judges statement-level correspondence, only among those candidates; and proof completeness comes from a deterministic transitive dependency audit never exposed to the model. Fusion assigns each item one of five coverage states, from Formalized to Missing, each backed by inspectable evidence. On consensus annotations of 98 items from three Lean projects, with three runs of each of two judge models, LeanTrail assigns the correct coverage state to 91.8% (Gemini 3.8 Flash) and 92.5% (GPT-6 Sol) of items. 96-97 of the 98 states are identical across runs, and GPT-6 Sol never reports a Partial item as Formalized. Uncovered items form a natural task queue for formalization agents, anchored to the specification rather than the compiler.
Problem
Lean compiler feedback certifies that declarations typecheck, not that they faithfully capture a source paper's results. A repository can compile while central theorems are weakened, merely auxiliary, or transitively dependent on sorry.
Approach
LeanTrail extracts theorem-like items from the paper's LaTeX source and indexes Lean declarations, their dependency graphs, and optional Blueprint nodes. For each paper item, BM25 and identifier-similarity rankings are fused with reciprocal rank fusion to retrieve the top-20 candidates. A language model judges statement-level correspondence only among those candidates. A separate deterministic transitive audit checks for sorryAx dependencies, and a fusion step assigns each item one of five coverage states (Formalized, Statement only, Partial, Blueprint only, Missing).
Results
Evaluation used 98 consensus-annotated items from three Lean projects (FinEqs, APAP, Ringel), with three runs of each judge model. LeanTrail assigned the correct coverage state to 91.8% of items with Gemini 3.8 Flash and 92.5% with GPT-6 Sol. 96-97 of the 98 states were identical across runs, and retrieval recall at 20 was 100%. Uncovered items are exported as agent-ready formalization tasks.
| Project | Gemini 3.8 Flash | GPT-6 Sol |
|---|
| FinEqs | 100.0 | 85.7 |
| APAP | 87.4 | 82.8 |
| Ringel | 93.0 | 97.8 |
| Overall | 91.8 ± 1.0 | 92.5 ± 0.6 |
Coverage-state accuracy (%) per project, mean over three runs