← All papers
First page of Automated Testing and Repair for Verified Compilers Generated by a Coding Agent

Automated Testing and Repair for Verified Compilers Generated by a Coding Agent

Martin Rinard

cs.SE Jul 31, 2026 · v1
Tests and agentically repairs Axon, a Lean 4 verified compiler, including its Lean refinement proofs, certificate checker, and executable operational semantics.
We present an agent based automated testing and repair system for verified compilers that contain four kinds of code: verified code, checked code, unverified code, and specification. We present specialized defect detection techniques that exploit the structure present in such compilers. For each surfaced defect the system invokes a coding agent to repair the defect and validate the repair. We evaluate the system on the Axon compiler, a compiler completely generated by a coding agent operating under developer supervision. The compiler was validated during development on a small benchmark set (the Livermore benchmarks), raising the possibility that its coding agent reward hacked the compiler. We also evaluate the possibility that the repair system reward hacked the repairs and find no evidence of reward hacking in either the Axon compiler or the repairs. We present results that characterize the testing and repair effectiveness and discuss repair characteristics.

Verified compilers such as Axon, which was generated by a coding agent, still contain unverified parts: parsers, printers, the ASM operational semantics, and checked-but-unverified optimizations. These parts can hide defects. Because Axon was validated on a small benchmark set, its coding agent may also have reward hacked the compiler.

ACDC uses Axon's structure to run targeted tests. It compares the executable Lean ASM semantics against real hardware execution on random instruction sequences, tests the parser and printer, and collects certificate rejections from the verified certificate checker on randomly generated programs. Each surfaced defect is handed to a Claude Code repair agent, which modifies Lean definitions and correctness proofs and then validates the repair. Campaigns alternate 30-minute detect phases with repair phases until no new defects or rejections appear.

The repair agents repaired all surfaced defects in the printer, semantics, parser, passes, and checker, including some repairs that changed over a hundred lines of Lean proof. Each repair cost roughly $1 to $18 in tokens. No evidence of reward hacking was found in either Axon or the repairs.

DefectComponentDef LOC (+/-)Proof LOC (+/-)Cost
CmpimmPrinter+13/-30/0$2.54
ShiftSemantics+6/-6+2/-2$2.94
FcmpSem+Printer+21/-5+24/-24$6.71
Selected ASM semantics/printer repairs