Mechanism-level routing failure in LLMs over Lean-verified algebraic structures
Manuel Israel Cázares, Wenlin Zhang, Haobo Ma
cs.CL
Jul 5, 2026 · v1
TL;DR
Evaluates LLM mechanism-classification routing over the FiberRing Lean 4 algebraic corpus, using Lean-verified artifacts as anchored ground-truth labels.
Abstract
We present an empirical study of structural routing failure in large language models (LLMs) over a formally verified algebraic corpus. The task requires selecting the correct proof-mechanism label from a fixed closed template set for compact mathematical objects drawn from the FiberRing formalization in Lean 4, where each item is anchored to a Lean-verified artifact and assigned a label from the corresponding certificate family. Our central finding is a mechanism-level routing ceiling: under blind conditions, gpt-oss-120b achieves 80.3% template accuracy on 22 FiberRing items (n=66; temperature=0, seed=0), while Llama 3.3 70B reaches 68.2%. Exposing a mechanism-bearing Lean verdict/witness cue (Condition A2) raises accuracy to 90.9% and 81.8% – gaps of +10.6 and +13.6 pp termed cue-induced routing uplift. The dominant failure is a CRT-to-ring-equivalence misroute: gpt-oss-120b misroutes 7 of 12 CRT items (58.3%) blind, zero under A2. A cross-model dissociation in Llama is notable: verdict accuracy is identical in both conditions (95.5%), while template accuracy improves 13.6 pp – confirming that truth inference and proof-mechanism classification are separable capacities. A cross-corpus extension (Set B; 6 POM/CollisionKernel items, 72 evaluations) provides a small cross-module check: CRT-granularity compression reappears with different labels, and an inverse cross-model dissociation emerges. These findings extend the router hypothesis (Cazares 2026) to formal algebraic structures. The full pipeline, manifest, and results are at
https://github.com/bytepro-ai/fiber-routing-eval.
Problem
LLMs often fail formal reasoning tasks not from missing sub-skills but from misrouting problems to the wrong structural interpretation. It is unclear whether this routing ceiling persists on formally verified mathematical objects with machine-checked labels.
Approach
A benchmark (FiberRing Set A, 22 items; plus 6 POM/CollisionKernel Set B items) is extracted from an Automath/Omega Lean 4 formalization derived from x^2=x+1, each item anchored to a named Lean declaration and a fixed proof-mechanism label. Models must select the mechanism label under two conditions: blind routing (A1) and routing with a mechanism-bearing Lean verdict/witness cue (A2). gpt-oss-120b and Llama 3.3 70B are evaluated at temperature 0, seed 0, three runs each (336 total evaluations).
Results
Blind template accuracy reaches 80.3% (gpt-oss-120b) and 68.2% (Llama), rising to 90.9% and 81.8% with the Lean cue. The dominant failure is a CRT-to-ring-equivalence misroute (7 of 12 CRT items blind for gpt-oss-120b, zero under A2). Truth (verdict) inference and mechanism classification are separable: Llama verdict accuracy stays 95.5% in both conditions while template accuracy improves 13.6 pp.
| Model | Cond. | Tmpl acc. | Verd. acc. | ΔTmpl |
|---|
| gpt-oss-120b | A1 | 80.3% | 84.8% | +10.6 pp |
| gpt-oss-120b | A2 | 90.9% | 95.5% | |
| Llama 3.3 70B | A1 | 68.2% | 95.5% | +13.6 pp |
| Llama 3.3 70B | A2 | 81.8% | 95.5% | |
Template accuracy, verdict accuracy, and caveat hit rate by model and condition