Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
Burc Gokden
cs.LG
Aug 10, 2026 · v1
cs.CL
TL;DR
Selected proof cores, including the theorem that SDPA equals PLGA at G_LM=I, are machine-checked in Lean 4.
Abstract
The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at $G_{LM}=I$; $A_{LM}$ and $A_P$ are strictly entrywise positive, with Perron-Frobenius structure on $A_{LM}$; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of $10^{-6}$ and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within $5\times 10^{-5}$ per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.
Problem
Scaled dot-product attention (SDPA) uses a fixed bilinear form for query-key comparison. The paper studies Power Law Graph Attention (PLGA), which replaces this with a learned input-generated bilinear operator, and asks how PLGA relates to SDPA and whether its deductive outputs collapse at inference.
Approach
The architecture and its tensor chain are fully specified, with claims labeled as theorem, conditional theorem, measurement, or conjecture. Structural results are proved: PLGA contains SDPA exactly at G_LM=I, entrywise positivity with Perron-Frobenius structure, a commutant criterion for relative-position invariance under RoPE nonresonance, and a conditional inference-collapse theorem. Selected proof cores are formalized and machine-checked in Lean 4. Empirical invariance and cache-fidelity bounds are measured on released checkpoints.
Results
Deductive outputs are measured to be invariant across inputs with relative fluctuations of order 10^-6 and below. Block and sequential scoring select identical answers and agree on the TruthfulQA probability-mass metric within 5x10^-5 per item. The SDPA-containment and exact-cacheability results are machine-checked, while collapse remains conditional on measured invariance.
| Claim | Rests on | Status |
|---|
| SDPA = PLGA at G_LM=I | Thm. 5.4(i) | proved, machine-checked |
| Exact cache under invariance | Prop. 5.2(i), Thm. 5.4(ii) | proved (sufficiency only) |
| Observed invariance | experiments + App. D | empirical |
| Positional codimension | Prop. 4.1, Cor. 7.1 | conditional theorem |
Status of key claims and their formal basis