ReuF2F: Benchmarking Coding Agents on Formalized Undergraduate Research Problems
Zeyu Zheng, Yiyang Liu, Yuyang Zhao, Congmin Zheng, Jiachen Zhu, Jianian Chen, Kangning Zhang, Kayue Daniel Wong, Lingyu Yang, Rong Shan, Shihao Chen, Thomas Zhu, Wentao Zhang, Xiaopeng Li, Yan Hu, Yongxi Lin, Yuxiang Chen, Zhuoying Ou, Jianghao Lin, Jeremy Avigad, Prasad Tetali, Jingbang Chen, Sean Welleck
cs.AI
Oct 19, 2026 · v1
cs.CL cs.LG
TL;DR
Introduces a Lean 4 benchmark of 244 formalized REU open problems, with agent submissions checked by Comparator, the Lean kernel, and nanoda.
Abstract
As AI systems begin to resolve open research problems, reliable verification becomes increasingly important. Formal mathematics provides a trusted framework for mathematical reasoning, enabling agents to produce machine-checkable proofs while conducting mathematical exploration. Existing formal mathematics benchmarks often focus on known results or a small set of prominent open problems, which can become difficult to evaluate reliably once solutions are discovered and widely disseminated. Existing evaluation protocols are also not always aligned with how modern coding agents operate. We therefore introduce ReuF2F, a Lean 4 benchmark of 244 formalized open problems drawn from Research Experiences for Undergraduates (REU), spanning entry-level open research problems across a broad range of mathematical subfields. We expect these less prominent open problems, which tend to attract less attention and may remain unsolved for longer, to be better suited for held-out evaluation of end-to-end open problem resolution in Lean. We adopt a standard coding-agent evaluation setting, placing each problem and its logical negation into a Lean project in a Docker container. We also build Leanimum-agent on mini-SWE-agent to provide a unified harness for fair comparisons across models. Our evaluation finds that even the best-performing model resolves only 40% of the problems, leaving substantial room for improvement on formalized open research problems.
Problem
Existing formal mathematics benchmarks focus on known results or a few prominent open problems. These are prone to leakage once the problems are solved. Their evaluation protocols also do not match how modern coding agents operate.
Approach
The authors build reuF2F, a set of 244 open conjectures drawn from Research Experiences for Undergraduates programs and formalized in Lean 4 with Mathlib. Each problem and its logical negation are placed in a Lean project inside a Docker container, so either a proof or a refutation counts as a resolution. All models run through a common harness, Leanimum-agent, built on mini-SWE-agent. Submissions pass layered checks: lake build, axiom checking, SafeVerify, Comparator kernel replay, and the independent kernel checker nanoda.
Results
The best model, Gemini 4 Argon (high effort), reaches 42.42% pass@1 and 44.67% pass@2. Performance increases with test-time reasoning effort, and weaker models resolve under 10% of problems.
| Model | Effort | pass@1 | pass@2 | proved@2 | refuted@2 |
|---|
| Gemini 4 Argon | high | 42.42 | 44.67 | 25.00 | 19.67 |
| Claude Opus 5.5 | max | 42.01 | 43.85 | 23.36 | 20.49 |
| GPT-6.1 Sol | max | 39.55 | 40.98 | 19.26 | 21.72 |
| GPT-6 Astra | max | 34.63 | 37.70 | 17.21 | 20.49 |
| Kimi K3 | max | 7.17 | 9.84 | 2.87 | 6.97 |
Top models on reuF2F (percent)