AutoAssumption: Searching over Assumptions Discovers ML-Systems Algorithms
Igor Sadalski, Minghao Guo, Benjamin Jones, Wojciech Matusik
cs.AI
Nov 10, 2026 · v1
cs.LG
TL;DR
LLM agents formalize 89 ML-systems algorithms as Lean-checked proof graphs; variants that relax one assumption must re-prove in Lean before benchmarking.
Abstract
Many advances in ML systems came from breaking an assumption that earlier methods took for granted: speculative decoding dropped the assumption that a model needs one forward pass per token. Yet algorithms rarely state their assumptions, so experts and discovery harnesses search over programs, where one edit changes several design choices and gains are difficult to attribute. Searching over the assumptions themselves remains largely unexplored. Writing algorithms as proofs makes their assumptions explicit and turns algorithm design into a search over a small, structured space that an LLM can explore and a proof checker can audit. We formalize 89 algorithms from KV-cache eviction, weight pruning and speculative decoding as proof graphs with explicit assumptions, checked in Lean. Our harness, AutoAssumption, relaxes one assumption of a published method, re-derives and re-proves it, and generates its code. In KV-cache eviction and weight pruning, this search improved on every published algorithm we ran and produced state-of-the-art algorithms. The eviction algorithm has the highest RULER-4K score of the published methods we ran, and the pruning algorithm the lowest perplexity of thirteen on four LLaMA models. At equal budget, the harness scores within a point of OpenEvolve, an open-source implementation of AlphaEvolve, with a lower standard deviation across repeats. Future work will formalize more fields and combine them through the principal assumptions they share.
Problem
Advances in ML-systems algorithms often come from breaking an implicit assumption. Code-level discovery harnesses edit programs, where a single edit changes several design choices at once, so gains are hard to attribute.
Approach
Each published algorithm is written as a proof graph of assumptions, definitions and theorems that ends in a formal problem statement (the contract). LLM agents produce the proof from the paper and code, transcribe it into Lean, and check it completely, along with a coverage test mapping code lines to proof statements. The AutoAssumption harness relaxes one pooled assumption of a parent algorithm, re-proves only the dependent statements in Lean, generates code, and benchmarks the variants that pass.
Results
89 algorithms were formalized: 44 in KV-cache eviction, 26 in speculative decoding and 19 in weight pruning. Starting from 26 parents, the search found a higher-scoring variant for every one. RestoreKV GM reached the best RULER-4K score among the published eviction methods run (77.0 vs 73.9 for its parent on Llama-3.1-8B), and Pruner-Zero LG had the lowest perplexity of thirteen pruning methods on four LLaMA models. At equal budget the harness scored within a point of OpenEvolve, with lower variance across repeats.
| Method | RULER 4K Llama | RULER 8K Llama | RULER 16K Llama | RULER 4K Qwen3 | LongBench Llama |
|---|
| RestoreKV | 73.9 | 74.8 | 73.3 | 72.1 | 37.8 |
| RestoreKV GM (ours) | 77.0 | 77.9 | 76.5 | 74.5 | 39.2 |
RestoreKV vs. RestoreKV GM: RULER and LongBench scores