ALPS: Measuring Valid Creativity in Large Language Models with Mathematical Construction
Eric Xie, Wenqian Ye, Aidong Zhang
cs.AI
Aug 17, 2026 · v1
TL;DR
Benchmark judge verifies trivial-side submissions as Lean proofs checked by the Lean kernel, with an autoformalization harness assembling Lean proofs from LLM equational derivations.
Abstract
Large language models produce outputs presented as discoveries - new proofs, conjectures, or molecules. Whether such an output that appears creative is truly original and effective is hard to establish: open-ended outputs require subjective judgment, the output may replicate something seen in training, or the task may be too simple to need creativity. We present ALPS (Austin-Law Proof-Synthesis), a benchmark that designs a task to measure valid creativity: producing a solution that is original and can be proven correct. Each instance is a single equational law, certified to require either the construction of an infinite mathematical structure satisfying the law, or a proof that no such structure exists. Submissions are verified by automated proof checking with no human involvement, and a public generator produces new instances without limit, so LLMs are never evaluated on problems they may have seen. A portfolio of eight configurations of leading automated provers resolves 2.2% of the 4,141-law evaluation pool, and a twentyfold budget increase adds 0.6%: the obstacle is not compute, but the absence of any method that produces the tailored structure each law requires. Under a fixed protocol, the strongest reasoning model we test succeeds in 14% of instances on the proof side, but none on the construction side. The remaining 97.2% of the pool is unresolved at every configuration and budget we test. We release ALPS in full: the corpus, the generator, and the automated judge.
Problem
Judging whether LLM outputs are genuinely creative is hard: open-ended answers need subjective judgment, may be memorized from training data, or come from tasks too easy to require creativity. The goal is a benchmark that is verifiable, renewable, and constructive at once.
Approach
ALPS instances are single magma equational laws of the form x = T, each certified by machine-checked proof to have no nontrivial finite model. Each law therefore either entails x=y or has an infinite nontrivial model (an Austin law). Trivial-side answers are judged as Lean proofs checked by the Lean kernel, assembled by a harness from LLM-emitted chains of equalities. Construction-side answers are judged by automated-prover saturation certificates. A public generator produces new instances without limit.
Results
Eight configurations of Vampire, E, and Twee resolve 2.2% of the 4,141-law residual pool, and a twentyfold budget increase adds only 23 triviality proofs and no new models. The strongest LLM tested (o3) succeeds on up to 14% of the certified-easy trivial instances and constructs no models on the hard tier. 97.2% of the pool remains unresolved.
| Budget (s) | New models | New trivial | Unresolved |
|---|
| 30 | 4 | 87 | 4,050 |
| 60 | 0 | 8 | 4,042 |
| 120 | 0 | 5 | 4,037 |
| 300 | 0 | 4 | 4,033 |
| 600 | 0 | 6 | 4,027 |
Portfolio resolutions on the residual pool by per-configuration budget