Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
Moritz Firsching, Paul Lezeau, Salvatore Mercuri, Miklós Z. Horváth, Yaël Dillies, Calle Sönne, Eric Wieser, Fred Zhang, Thomas Hubert, Blaise Agüera y Arcas, Pushmeet Kohli
cs.AI
May 13, 2026 · v1
TL;DR
Presents Formal Conjectures, an evolving benchmark of 2615 research-level problems formalized in Lean 4.
Abstract
As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their capabilities. To address this, we present Formal Conjectures, an evolving benchmark of currently 2615 mathematical problem statements formalized in Lean 4. Sourced from areas of active mathematical research, the dataset features 1029 open research conjectures providing a zero-contamination benchmark for mathematical proof discovery, and 836 solved problems for proof autoformalization. Notably, the repository provides a structured interface connecting mathematicians who formalize and clarify problems with the AI systems and humans attempting to solve them. Demonstrating its immediate utility, the benchmark has already been leveraged to make new mathematical discoveries, including the resolution of open research conjectures. We describe our approach to ensuring the correctness of these formalizations in a collaborative open-source project where contributions stem from an active community. In this framework, AI-generated proofs and disproofs serve as a valuable auditing mechanism to iteratively improve the fidelity of the benchmark. Finally, we provide a standardized evaluation setup and report baseline results on frozen evaluation subsets, demonstrating a climbable signal that measures the current frontier of automated reasoning on research-level mathematics.
Problem
Existing benchmarks for automated mathematical reasoning suffer from data leakage, secretive evaluation, oversimplified success criteria, and saturation. There is a need for research-level formal problems that provide zero-contamination evaluation of genuine reasoning.
Approach
Formal Conjectures is an evolving benchmark of 2615 mathematical problem statements formalized in Lean 4 using Mathlib. It sources problems from active mathematical research, including 1029 open research conjectures (zero-contamination) and 836 solved problems for proof auto-formalization. The repository provides a structured interface connecting mathematicians who formalize problems with AI systems attempting to solve them.
Results
The benchmark has already been used to resolve open research conjectures. Baseline results on frozen evaluation subsets show climbable signal: AlphaProof achieves 45-50% on the test subset, and a DM prover agent reaches 66%. AI-generated proofs serve as an auditing mechanism to improve formalization fidelity.
| Method | Proved (%) |
|---|
| AlphaProof (1k sims) | 45.0% |
| AlphaProof (16k sims) | 50.0% |
| DM prover agent (dev) | 66.0% |
Baseline results on frozen evaluation subset