← All papers
First page of ReuF2F: Benchmarking Coding Agents on Formalized Undergraduate Research Problems

ReuF2F: Benchmarking Coding Agents on Formalized Undergraduate Research Problems

Zeyu Zheng, Yiyang Liu, Yuyang Zhao, Congmin Zheng, Jiachen Zhu, Jianian Chen, Kangning Zhang, Kayue Daniel Wong, Lingyu Yang, Rong Shan, Shihao Chen, Thomas Zhu, Wentao Zhang, Xiaopeng Li, Yan Hu, Yongxi Lin, Yuxiang Chen, Zhuoying Ou, Jianghao Lin, Jeremy Avigad, Prasad Tetali, Jingbang Chen, Sean Welleck

cs.AI Oct 19, 2026 · v1 cs.CL cs.LG
Introduces a Lean 4 benchmark of 244 formalized REU open problems, with agent submissions checked by Comparator, the Lean kernel, and nanoda.
As AI systems begin to resolve open research problems, reliable verification becomes increasingly important. Formal mathematics provides a trusted framework for mathematical reasoning, enabling agents to produce machine-checkable proofs while conducting mathematical exploration. Existing formal mathematics benchmarks often focus on known results or a small set of prominent open problems, which can become difficult to evaluate reliably once solutions are discovered and widely disseminated. Existing evaluation protocols are also not always aligned with how modern coding agents operate. We therefore introduce ReuF2F, a Lean 4 benchmark of 244 formalized open problems drawn from Research Experiences for Undergraduates (REU), spanning entry-level open research problems across a broad range of mathematical subfields. We expect these less prominent open problems, which tend to attract less attention and may remain unsolved for longer, to be better suited for held-out evaluation of end-to-end open problem resolution in Lean. We adopt a standard coding-agent evaluation setting, placing each problem and its logical negation into a Lean project in a Docker container. We also build Leanimum-agent on mini-SWE-agent to provide a unified harness for fair comparisons across models. Our evaluation finds that even the best-performing model resolves only 40% of the problems, leaving substantial room for improvement on formalized open research problems.

Existing formal mathematics benchmarks focus on known results or a few prominent open problems. These are prone to leakage once the problems are solved. Their evaluation protocols also do not match how modern coding agents operate.

The authors build reuF2F, a set of 244 open conjectures drawn from Research Experiences for Undergraduates programs and formalized in Lean 4 with Mathlib. Each problem and its logical negation are placed in a Lean project inside a Docker container, so either a proof or a refutation counts as a resolution. All models run through a common harness, Leanimum-agent, built on mini-SWE-agent. Submissions pass layered checks: lake build, axiom checking, SafeVerify, Comparator kernel replay, and the independent kernel checker nanoda.

The best model, Gemini 4 Argon (high effort), reaches 42.42% pass@1 and 44.67% pass@2. Performance increases with test-time reasoning effort, and weaker models resolve under 10% of problems.

ModelEffortpass@1pass@2proved@2refuted@2
Gemini 4 Argonhigh42.4244.6725.0019.67
Claude Opus 5.5max42.0143.8523.3620.49
GPT-6.1 Solmax39.5540.9819.2621.72
GPT-6 Astramax34.6337.7017.2120.49
Kimi K3max7.179.842.876.97
Top models on reuF2F (percent)