Builds a benchmark of 500 Lean 4 proof obligations extracted from 23 open-source Lean verification repositories and evaluates LLM provers on them.
Abstract
Large language models have achieved striking results in interactive theorem proving, particularly in Lean. However, most benchmarks for LLM-based proof automation are drawn from mathematics in the Mathlib ecosystem, whereas proofs in software verification are developed inside definition-rich codebases with substantial project-specific libraries. We introduce VeriSoftBench, a benchmark of 500 Lean 4 proof obligations drawn from open-source formal-methods developments and packaged to preserve realistic repository context and cross-file dependencies. Our evaluation of frontier LLMs and specialized provers yields three observations. First, provers tuned for Mathlib-style mathematics transfer poorly to this repository-centric setting. Second, success is strongly correlated with transitive repository dependence: tasks whose proofs draw on large, multi-hop dependency closures are less likely to be solved. Third, providing curated context restricted to a proof's dependency closure improves performance relative to exposing the full repository, but nevertheless leaves substantial room for improvement. Our benchmark and evaluation suite are released at https://github.com/utopia-group/VeriSoftBench.
Problem
Most Lean benchmarks for LLM proof automation are Mathlib-centric mathematics. Software verification proofs instead live in definition-rich repositories with project-specific libraries and cross-file dependencies, and it is unclear how well current provers handle that setting.
Approach
VeriSoftBench collects 500 Lean 4 theorems from 23 repositories covering compilers, PL semantics, type systems, verification frameworks and applied verification. Each task is pinned to a fixed commit with its original proof environment preserved. Tasks are filtered to have hole-free ground-truth proofs, to compile with the proof replaced by sorry, and to be non-trivial for standard automation. Frontier LLMs, Goedel-Prover-V2 and Aristotle are evaluated under two settings: curated dependency-closure context and full-repository context.
Results
Gemini-3-Pro is the strongest general model, solving 41.0% of tasks with curated context and 34.8% with full context. Goedel-Prover-V2 solves 0% in the full-repository setting. Success falls as transitive repository dependence grows, and curated context helps but leaves large headroom.
Figure 6 : Cumulative distribution of theorem dependency depth (log scale). Line shows mean across repositories; shading indicates ±1 std.