Spread Too Thin: How Higher Resolution Dilutes a Vision Transformer's Attention, and a Training-Free Rescaling
Neil D. B. Bruce
cs.CV
Nov 1, 2026 · v1
TL;DR
Every mathematical statement about softmax attention dilution and the log-ratio rescaling is proved in Lean 4 and released as ancillary files.
Abstract
A vision transformer read at a higher resolution than it was trained at spreads each attention row over many more patches, yet every row must still hand out exactly one unit of weight. A law of softmax attention predicts that a growing crowd of bounded scores dilutes the attention on any patch or register whose score stays fixed. The same law gives a remedy that needs no training: multiply every score, the sink's included, by the ratio of the logarithms of the test and training token counts. Given sufficient margins at the training resolution, this keeps the attention on a match when one is present, and on a sink such as a register when none is, though trained models need not meet those margins. We test these predictions in public vision transformers and in controlled models read far above their training token count. Attention spreads as predicted. Registers lose their share, though production models compensate in part, and the rescaling holds the share of DINOv2's registers nearly level. Applied at test time without training, the rescaling recovers part of the lost accuracy in nearly every model, with absolute or rotary positions and with registers, and on images with native detail. The calibrated multiplier rises with the logarithmic ratio on average and varies across models. Its rise is steeper than the square-root rule used in image generators; in controlled models the log ratio is also more accurate, while in the public models the two rules' accuracy difference remains unresolved. The rescaling does not restore the attention a model learned, and why it helps, beyond countering the dilution, remains open.
Problem
Vision transformers run at higher resolution than they were trained at spread each attention row over many more patches. This dilutes the attention on matching patches and on sinks such as registers, and accuracy drops sharply.
Approach
A softmax attention law predicts the dilution and gives a training-free remedy: multiply every score, the sink's included, by λ = ln N / ln N_tr. The predictions are tested on eleven public ImageNet-1k ViTs (absolute positions, rotary positions, and register models), on DINOv2 with registers, and on controlled ViT-S models, at up to 16× the training token count. Every mathematical statement is proved in a Lean 4 package provided as ancillary files.
Results
Attention spreads as predicted, and registers lose share; the rescaling keeps DINOv2's register share nearly level. At test time the rescaling recovers part of the lost accuracy in 10 of 11 checkpoints on enlarged crops and in all 11 on resized originals. The calibrated multiplier tracks λ on average (slope 0.945) and rises more steeply than the square-root rule.
| Model | Training-size top-1 | Crops: trained → rescaled | Gain |
|---|
| DeiT-S | 79.9 | 31.3 → 53.2 | +21.9 |
| DeiT-B | 82.0 | 59.7 → 67.1 | +7.4 |
| AugReg ViT-S | 81.4 | 48.5 → 61.4 | +12.9 |
| RoPE-Mixed ViT-B | 83.8 | 49.6 → 63.2 | +13.6 |
| AugReg2 ViT-B | 85.1 | 46.3 → 45.6 | −0.8 |
Top-1 at 16× training tokens on enlarged crops, as trained vs. rescaled (selected models)