← All papers
First page of GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization with Enhanced Training Data Utilization for Sample-Constrained Reinforcement Learning

GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization with Enhanced Training Data Utilization for Sample-Constrained Reinforcement Learning

Zhengqing Yan, Xinyang Liu, Yi Zhang, Fan Guo, ChengXun Jia, Junchen Wan, Yao Liu, Qi Liu, Jihao Huang, Kang Song

cs.AI Jan 11, 2026 · v1
Trains an LLM theorem prover with RL using the Lean verifier's binary correctness signal to gate sampling and decouple advantage sign from auxiliary rewards.
Automated Theorem Proving (ATP) represents a fundamental challenge in Artificial Intelligence (AI), requiring the construction of machine-verifiable proofs in formal languages such as Lean to evaluate AI reasoning capabilities. Reinforcement learning (RL), particularly the high-performance Group Relative Policy Optimization (GRPO) algorithm, has emerged as a mainstream approach for this task. However, in ATP scenarios, GRPO faces two critical issues: when composite rewards are used, its relative advantage estimation may conflict with the binary feedback from the formal verifier; meanwhile, its static sampling strategy may discard entire batches of data if no valid proof is found, resulting in zero contribution to model updates and significant data waste. To address these limitations, we propose Group Dual-dynamic and Equal-right-advantage Policy Optimization (GDEPO), a method incorporating three core mechanisms: 1) dynamic additional sampling, which resamples invalid batches until a valid proof is discovered; 2) equal-right advantage, decoupling the sign of the advantage function (based on correctness) from its magnitude (modulated by auxiliary rewards) to ensure stable and correct policy updates; and 3) dynamic additional iterations, applying extra gradient steps to initially failed but eventually successful samples to accelerate learning on challenging cases. Experiments conducted on three datasets of varying difficulty (MinF2F-test, MathOlympiadBench, PutnamBench) confirm the effectiveness of GDEPO, while ablation studies validate the necessity of its synergistic components. The proposed method enhances data utilization and optimization efficiency, offering a novel training paradigm for ATP.

In automated theorem proving with reinforcement learning, GRPO faces two issues: its relative advantage estimation may conflict with binary verifier feedback when composite rewards are used, and its static sampling strategy discards entire batches when no valid proof is found, wasting data.

GDEPO introduces three mechanisms: (1) dynamic additional sampling, which resamples invalid batches until a valid proof is discovered; (2) equal-right advantage, decoupling the sign of the advantage (based on correctness) from its magnitude (modulated by auxiliary rewards); and (3) dynamic additional iterations, applying extra gradient steps to initially failed but eventually successful samples. The system uses Lean 4 as the formal verifier.

Experiments on MiniF2F-test, MathOlympiadBench, and PutnamBench confirm the effectiveness of GDEPO over GRPO. Ablation studies validate that all three mechanisms contribute synergistically to improved data utilization and optimization efficiency.