← All papers
First page of Learn then Bet: Learning Interpretable Rules for Reliable Agent Intervention

Learn then Bet: Learning Interpretable Rules for Reliable Agent Intervention

You-Lin Chen, Xinyi Lisa Chen, Pedro Herrero-Vidal, Xiaojing Yu, Bingqing Song, Bin Xu

cs.AI Nov 13, 2026 · v1 cs.LG
The paper's statistical guarantees are machine-checked in Lean 4 with Mathlib measure theory. The proofs were AI-generated, and Comparator checks that the Lean claims match the paper's statements.
Autonomous agents now run for hundreds of steps with little or no human supervision. In practice, practitioners use runtime monitors such as turn limits and keyword filters to decide when a run should be stopped, upgraded, or assisted. However, these heuristics may stop runs that would otherwise succeed or allow failing runs to continue. To address this limitation, we introduce LeBet (Learn then Bet), a statistical learning framework that derives its learning objective from sufficient conditions for valid and powerful intervention. In particular, LeBet learns an interpretable OR-of-AND monitor and accumulates evidence through a fixed betting statistic, intervening when the evidence reaches a prescribed threshold. Under finite within-class exchangeability and leave-one-out stability, we derive finite-sample guarantees for false-positive control (validity) and detection power conditional on the training data, using the same trajectories for learning and calibration without assuming independent and identically distributed trajectories. To address the intractable combinatorial optimization problem in our learning formulation, we use stochastic local search that leverages the monitor's Boolean structure to propose rule edits aimed at improving empirical separation. On SWE-bench Lite with Ministral 3 8B, the learned monitor saves $34.46\%$ of input-plus-output tokens and increases output precision from $18.33\%$ to $21.59\%$. The empirical false-positive rate is $13.64\%$. Our results show how to formalize the learning problem over interpretable rule sets with statistical conditions and establish training-conditional guarantees for reliable intervention based on partial histories.

Long-running LLM agents are usually stopped or assisted by heuristic runtime monitors such as turn limits and keyword filters. These monitors may halt runs that would succeed or let failing runs continue, and they come without statistical guarantees.

LeBet learns an interpretable OR-of-AND rule monitor over adapted binary predicates of the trajectory. Evidence is accumulated with a fixed betting statistic, and the agent is interrupted when that evidence crosses a threshold. The learning objective is an empirical separation derived from sufficient conditions for false-positive control and detection power, and it is optimized by stochastic local search over rule edits. Finite-sample, training-conditional guarantees are proved under within-class exchangeability and leave-one-out stability. The definitions and theorem statements are encoded as Lean 4 Challenges, with AI-generated Solutions checked by Lean and validated with Comparator.

On SWE-bench Lite with Ministral 3 8B, the learned monitor saves 34.46% of input-plus-output tokens and raises output precision from 18.33% to 21.59%, with an empirical false-positive rate of 13.64%. The main guarantees are machine-checked in Lean 4.