← All papers
First page of Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head Attention

Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head Attention

Karthik Viswanathan, Plamen Dimitrov, Stefano Rocca, Sarayu Sirikonda

cs.CC Oct 18, 2026 · v1 cs.AI cs.LG
All results on multi-head attention head complexity are formalized and machine-checked in Lean 4 with Mathlib, with the formalization generated by an AI agent.
Multi-head attention is a central component of the transformer architecture, yet how its expressivity depends on the number of heads remains poorly understood. We study a one-layer, attention-only transformer with additive positional embeddings and define the head complexity $H^\ast(f)$ of a Boolean function $f$ as the minimum number of attention heads required to represent it. We derive an exact linear-fractional normal form for head complexity, prove a general threshold-degree lower bound, and give an explicit ten-bit function for which this bound is strict. For symmetric Boolean functions, we independently show that head complexity equals the number of times the function changes value between consecutive Hamming-weight levels, even when positional embeddings are allowed. This agrees with the concurrent characterization of Hsu and Xu (arXiv:2608.04243) in a model without positional embeddings. The symmetric lower bound combines our general threshold-degree bound with the classical result of Aspnes et al. (1994). These results were discovered and formalized by a frontier model through conjecture generation, theorem proving, and machine verification, while human researchers asked the high-level questions and reviewed the generated theorems. We make the Lean formalization and research traces publicly available to support agent-assisted theorem proving and formal verification in mechanistic interpretability. This work provides a proof of concept for using the capabilities of frontier models, grounded by formal verification, to develop an understanding of transformer architectures.

How the expressivity of multi-head attention depends on the number of heads is poorly understood. The work defines the head complexity H*(f) of a Boolean function as the minimum number of heads a one-layer attention-only transformer with positional embeddings needs to represent it exactly.

A frontier model agent (GPT-5.6 Sol in a Codex sandbox) generated conjectures and natural-language proofs over about 9.5 hours. A separate counterexample search produced a separating example. The agent was then prompted to formalize the results in Lean 4 using Mathlib. Humans posed the questions, checked that the Lean statements matched the intended claims, and refactored the code.

The results are an exact linear-fractional normal form for head complexity, a threshold-degree lower bound, and an explicit ten-bit function on which that bound is strict. For symmetric functions, head complexity equals the number of value changes across Hamming-weight levels. All results are machine-checked in Lean, and the formalization and research traces are released.