← All papers
First page of Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

Burc Gokden

cs.LG Aug 10, 2026 · v1 cs.CL
Selected proof cores, including the theorem that SDPA equals PLGA at G_LM=I, are machine-checked in Lean 4.
The Large Language Model from Power Law Decoder Representations (PLDR-LLM) and its attention, Power Law Graph Attention (PLGA), replace the fixed bilinear form of scaled dot-product attention (SDPA) with a learned, input-generated bilinear operator $G_{LM}$, built from a positive tensor $A_{LM}$ by elementwise power laws. The architecture is fully specified, verified against pinned reference releases; claims are labeled theorem, conditional theorem, measurement, or conjecture. Unconditionally: PLGA contains SDPA exactly at $G_{LM}=I$; $A_{LM}$ and $A_P$ are strictly entrywise positive, with Perron-Frobenius structure on $A_{LM}$; the DAG regularizer has the NOTEARS walk-counting form and positivity obstructs exact acyclicity; and, under nonresonance (satisfied by standard rotary frequencies), a commutant criterion identifies which operators preserve relative-position dependence. An inference-collapse theorem: exact input invariance of deductive outputs collapses inference to generalized SDPA with a constant operator. Measured invariance: relative fluctuations of $10^{-6}$ and below; perturbation bounds quantify but do not certify cached inference; the assembled proxy misses the decoding margin. A conditional three-stage mechanism (rotary twirl, concentration, row-map contraction) is measured on a released checkpoint. Blockwise training and scoring under the global Gram are stated with explicit target exposure; on tested samples, block and sequential scoring select identical answers and agree on the published TruthfulQA probability-mass metric within $5\times 10^{-5}$ per item. Self-organized criticality enters as a phenomenological framework with an intrinsic order parameter; open claims become falsifiable conjectures. Selected proof cores are machine-checked in Lean 4.

Scaled dot-product attention (SDPA) uses a fixed bilinear form for query-key comparison. The paper studies Power Law Graph Attention (PLGA), which replaces this with a learned input-generated bilinear operator, and asks how PLGA relates to SDPA and whether its deductive outputs collapse at inference.

The architecture and its tensor chain are fully specified, with claims labeled as theorem, conditional theorem, measurement, or conjecture. Structural results are proved: PLGA contains SDPA exactly at G_LM=I, entrywise positivity with Perron-Frobenius structure, a commutant criterion for relative-position invariance under RoPE nonresonance, and a conditional inference-collapse theorem. Selected proof cores are formalized and machine-checked in Lean 4. Empirical invariance and cache-fidelity bounds are measured on released checkpoints.

Deductive outputs are measured to be invariant across inputs with relative fluctuations of order 10^-6 and below. Block and sequential scoring select identical answers and agree on the TruthfulQA probability-mass metric within 5x10^-5 per item. The SDPA-containment and exact-cacheability results are machine-checked, while collapse remains conditional on measured invariance.

ClaimRests onStatus
SDPA = PLGA at G_LM=IThm. 5.4(i)proved, machine-checked
Exact cache under invarianceProp. 5.2(i), Thm. 5.4(ii)proved (sufficiency only)
Observed invarianceexperiments + App. Dempirical
Positional codimensionProp. 4.1, Cor. 7.1conditional theorem
Status of key claims and their formal basis