Humanize: Judgement Engineering for Agentic Coding
Sihao Liu, Ligeng Zhu, Zijian Zhang, Dongyun Zou, Zhengyang Zhang, Changye Li, Song Bian, Song Han, Tony Nowatzki
cs.AI
Oct 6, 2026 · v2
cs.CY
TL;DR
Applies a builder–reviewer agent loop to produce Lean proofs checked by the Lean kernel, scoring 672/672 on PutnamBench, 251/303 on Lean-Eval, and solving IMO 2026 formally.
Abstract
Agentic coding makes code generation cheap, but reliable completion remains difficult: the agent that writes the code is a weak judge of whether it is done. We present Humanize, a multi-agent orchestration workflow for agentic coding built around judgement engineering: explicit, mechanically enforced decisions at the boundaries between planning, implementation, review, and learning. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from another vendor decides completion; deterministic hooks, not a model, route work between these roles and enforce 72 mechanical gates. Viewed as a Markov chain over repository states, alternating builder and reviewer samples jointly from two models, so a defect survives only if both miss it. We study Humanize through its deployment, 118 public postmortems of real loops, and its applications. Over 68 versions in 108 days, it gathered 1,468 GitHub stars. Applications include a 567-file gem5 build-system migration under upstream review; Kernel Design Agents, which extend the loop with a kernel knowledge base and profiling feedback and placed in the top three of all three Full-Agent tracks of the MLSys 2026 FlashInfer contest; and, through Humanize Olympiad Agents (HOA), full scores in IOI 2026, IMO 2026, IPhO 2026, and IBO 2024, 418.5/437 in IChO 2026 (gold-medal). Humanize also achieves 672/672 on PutnamBench and ranks first (251/303) on Lean-Eval's leaderboard even competiting with professional mathematicians. The postmortems show that independent review catches unsupported builder claims, but stopping remains a key weakness. In reports that separate rounds by phase, two thirds of rounds occurred after implementation was accepted. This evidence is observational, not a controlled comparison of workflows.
Problem
In agentic coding, generating code is cheap, but deciding when work is actually complete is hard. The agent that writes the code is a weak judge of its own output.
Approach
Humanize is a multi-agent orchestration workflow. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from a different vendor decides completion. Deterministic hooks route work between these roles and enforce 72 mechanical gates. For formal proofs, the Lean kernel and a statement-preservation check run before the reviewer, which must also obtain approval from an independent proof verifier.
Results
Humanize scores 672/672 on PutnamBench and ranks first (251/303) on the Lean-Eval legacy leaderboard. It proved all six IMO 2026 problems in Lean from AxiomProver's formal statements. Other applications include a gem5 build-system migration, kernel design agents, and olympiad agents; 118 public postmortems indicate that stopping remains a weakness.