← All papers
First page of From Verification Failures to Reusable Guidance for Coding Agents

From Verification Failures to Reusable Guidance for Coding Agents

Yuqing Zhai, Xiaohong Chen, Lingming Zhang, Sriram Vishwanath, Grigore Rosu

cs.SE Sep 30, 2026 · v3 cs.AI cs.LO cs.PL
Residual mathematical domain lemmas from K-framework proofs are exported by the klean tool as Lean 4 theorems, then proved by agents and kernel-checked.
Coding agents need to establish that a program satisfies a specification and that the specification captures the requested behavior. We study how expert diagnosis of verification failures can become reusable guidance for this work. Our approach combines executable language definitions in the K framework with a kit of procedures for constructing specifications, repairing proofs, and auditing their adequacy. A human-guided development campaign on HumanEval, a benchmark of 164 Python programming tasks, achieves a 164/164 success rate with the semantics and the kit, measured by final AI audit Pass verdicts after two targeted repairs. To examine whether auditing detects problems that successful proofs leave unresolved, we construct 12 author-reviewed pairs of clean and defective packages. Every package passes its K proofs, and completed audits identify all defects and accept all clean packages. We then use KleverBench to test specification and proof construction for 31 programs with changed operator meanings. Comparisons with complete acceptance rules and equally long generic advice yield mixed results across two model and budget settings, motivating further work on selecting useful guidance within resource limits. Human-reviewed Optimism proofs establish expected pause reverts for six operations within declared input bounds under London semantics with unbounded gas. We report progress, difficulties, and lessons toward agents that deliver programs with checkable correctness arguments.

Coding agents need to show both that a program satisfies a specification and that the specification captures the requested behavior. Lessons from diagnosing verification failures are usually not retained for later agents.

The authors combine executable language semantics in the K framework with a curated verification kit. The kit holds procedures for building specifications, repairing proofs, and auditing adequacy. Experts diagnose failures and record revised semantic rules or procedures, which agents use at inference time. In the HumanEval workflow, K handles program reasoning, and residual domain lemmas are exported via klean to Lean 4, where agents prove them and Lean checks the proofs.

A human-guided campaign on HumanEval reaches 164/164 final AI audit Pass verdicts after two targeted repairs, one of which adds a Lean theorem. Audits detect all defects in 12 clean/defective package pairs that all pass their K proofs. On KleverBench, comparisons of the kit against complete acceptance rules and generic advice give mixed results across two models; separate smart-contract case studies (DSToken and others, Optimism) are also reported.

Semantics / kitPassConcernsFailPass rateLegit rate
Agent-authored / no kit234110014.0%39.0%
Supplied / no kit37369122.6%44.5%
Supplied / kit9767059.1%100.0%
Historical HumanEval audits at Stage 2 (164 tasks per arm)
Defect categoryOriginalAfter retriesClean accepted
Weak postcondition3/33/33/3
Unsupported bridge2/33/33/3
Source/proof mismatch3/33/33/3
Incorrect semantics2/33/33/3
Total10/1212/1212/12
Audit challenge: defects detected across 12 clean/defective pairs