← All papers
First page of From Reasoning Strings to Partial Orders: Verifier-Certified Rule Transport through Quotient Policy Optimization

From Reasoning Strings to Partial Orders: Verifier-Certified Rule Transport through Quotient Policy Optimization

Bang Xie, Hao Liu, Zhiyuan Peng, Xin Yin, Chenhao Ying, Yuan Luo, Senjian Zhang, Wei Chen

cs.LG Aug 20, 2026 · v1
Lean is one of three environments; its kernel replays proof steps to certify whether independent sibling subgoals commute, supplying RL training signal.
Many computations admit several valid execution orders because independent subgoals or disjoint state updates can commute. Reinforcement learning with verifiable rewards usually treats each successful trace as a separate token sequence, so serialization choices can be mistaken for logical dependencies. We introduce Verifier-Certified Rule Transport (VCRT), which replays adjacent operation pairs with native verifiers. Pairs whose two orders are accepted and reach the same canonical state provide commutation certificates; rejected or state-changing reversals provide anti-diamonds. VCRT uses anti-diamonds to preserve genuine prerequisites and assigns policy credit to the total probability mass of each certified orbit. It also constrains post-swap consistency, source retention, and policy drift. We evaluate leave-one-environment-out transfer across ProofWriter, CLRS, and Lean through a shared anonymized relation-graph interface. All training and checkpoint decisions are frozen before held-out evaluation, which uses one greedy trajectory per item without search or verifier feedback. VCRT obtains a 77.60% macro pass rate versus 64.53% for the strongest matched baseline, a paired gain of 13.06 points (95% bootstrap CI [12.58, 13.54]). Lean accounts for most of this gain at 33.49 points, while ProofWriter and CLRS improve by 2.85 points on average. Mechanism tests consistently favor anti-diamond supervision, whereas No-Orbit is statistically indistinguishable from full VCRT. The evidence does not establish a general benefit from exact orbit aggregation.

Reinforcement learning with verifiable rewards treats each successful trace as a separate token sequence. Harmless serialization choices can therefore be mistaken for logical dependencies. Indiscriminate order augmentation has the opposite flaw: it can reward invalid permutations.

Verifier-Certified Rule Transport (VCRT) replays adjacent operation pairs in both orders with native verifiers (a ProofWriter rule engine, CLRS state transitions, and the Lean kernel). Pairs that are accepted in both orders and reach the same canonical state yield commutation certificates. Rejected or state-changing reversals yield anti-diamonds. Policy credit is assigned to the total probability mass of certified orbits, combined with an anti-diamond margin loss and consistency, retention, and KL terms. Evaluation is leave-one-environment-out through an anonymized relation-graph interface, using one greedy trajectory per item.

VCRT reaches a 77.60% macro pass rate versus 64.53% for Canonical-GRPO, a paired gain of 13.06 points (95% CI [12.58, 13.54]). Most of the gain comes from Lean, which reaches 100%, possibly a ceiling effect. Anti-diamond supervision helps in every fold, while removing orbit aggregation makes no significant difference.

MethodHeld-Out LeanHeld-Out ProofWriterHeld-Out CLRSMacro
Outcome-GRPO57.3554.6961.1257.72
Canonical-GRPO66.5155.8471.2564.53
VCRT100.0057.9574.8477.60
Held-out pass rate (%) with one greedy trajectory per item (mean of three runs)