← All papers
First page of Quantitative Comparison of Credible Compilation and Verification In Coding Agent Compiler Development

Quantitative Comparison of Credible Compilation and Verification In Coding Agent Compiler Development

Martin Rinard

cs.PL May 9, 2026 · v1
A coding agent implements and proves correct compiler optimizations in Lean 4 within the Axon verified compiler, comparing proof effort against credible compilation.
Formal program verification is a longstanding goal in the field. We present the first quantitative comparison of the two primary compiler verification approaches, credible compilation/translation validation and full verification. Working with the first verified compiler developed by a coding agent (operating under human supervision), we present quantitative results from a coding agent implementing several optimizations using these two approaches. The results indicate that 1) verification requires roughly an order of magnitude more development effort than credible compilation, 2) to enhance provability, the coding agent chooses less efficient algorithms and data structures for verified optimizations, and 3) in an attempt to minimize proof effort the coding agent repeatedly implemented optimization scope reductions for verified optimizations, and 4) certificate checking time dominates optimization and certificate generation time for the considered optimizations. Because of the increased proof overhead, verified optimizations required substantially more supervision and coding sessions than credible compilation optimizations. Given the capabilities of a modern coding agent working in this context, implementation efforts for both credible compilation and verified versions remained feasible for the considered optimizations (unreachable code elimination, dead assignment elimination, and constant propagation/folding).

Compiler correctness can be ensured either by credible compilation (translation validation with a verified checker) or by full verification of each transformation. No quantitative comparison of the engineering effort the two approaches require has existed.

Starting from the Axon compiler written in Lean 4, a human supervised Claude Code as it implemented credible-compilation and fully verified versions of three optimizations: unreachable code elimination, dead assignment elimination, and constant propagation/folding. The agent wrote all code, certificate generators, and correctness proofs. Effort was measured by sessions, active time, tokens, code and proof lines of code, and supervisor prompts. Compile times were benchmarked on the Livermore kernels.

Verification took roughly an order of magnitude more effort than credible compilation, and the verified versions needed substantially more sessions and supervision. To ease proofs, the agent chose less efficient algorithms and data structures and repeatedly narrowed optimization scope. Certificate checking time dominated optimization and certificate generation time.

OptimizationSessionsTokensCode LoCProof LoC
UCE+DAE CC41.86 M+287, -56+0, -0
CP CC31.75 M+149, -24+0, -0
UCE VF812.27 M+549, -190+4,348, -290
UCE+DAE VF1416.35 M+734, -301+8,158, -662
CP VF1517.86 M+1,079, -170+6,131, -1,244
Engineering effort per optimization (CC = credible compilation, VF = verified)