RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill
Dylan Johnston, Robert P. Smith
cs.LG
Nov 6, 2026 · v1
cs.CL
TL;DR
Supporting statements listed in an appendix table are proved in Lean 4 against Mathlib, in the KvaLean project shipped in the data repository.
Abstract
RidgeFill shortens prefill by not computing most of a model's upper layers for the prompt. The lower layers run exactly up to a split; from the split layer's state a per-layer map predicts every later layer's input, and each late layer writes its cache from that prediction. The design follows YOCO and DeepSeek-V4.1-Flash, where the projection is trained into the model. RidgeFill retrofits it onto models that were never trained for it: the map is a ridge regression fitted once in closed form from one pass over calibration text, the layer's own projections run unchanged, and a chosen set of rows, the last few thousand tokens, and every decoding step run the full model. Nothing is trained and no token is dropped. Most of the work was in measurement, finding a repeatable methodology. Loss averaged over all tokens hides RidgeFill's error. Needle retrieval in the filled region, loss on tokens that depend on distant context, and splice tests that swap filled keys or values for exact ones show which component carries it: the keys on Llama-3.1-8B, half of it in their scale, and the values and the recurrent state on Qwen3.5-9B. On Qwen3.8-27B in a llama.cpp fork on an A100, time to first token is 1.25 to 1.31x faster at 16K to 120K tokens with the maps in VRAM; RULER changes by -1.2 and -1.4 points and LongBench v2 by -1.4; the 95 % intervals include zero but do not rule out losses of a few points. On Qwen3.8-Flash-Next in vLLM it is 1.50 to 1.60x faster than exact at 9K to 32K tokens, with RULER's needle tasks at 99.8 against 100.0 and perplexity on novel tokens 2.65 % higher [-0.02 %, +4.93 %]. On three 8 to 9B reference models, at the same split and tail as the identity fill, the fitted map retrieves 49.1 and 49.6 more filled-region RULER points on the two dense models (at that split Llama's loss is worse), and its long-document loss is +/-0.03 nats of the released SwiftKV checkpoint's.
Problem
Prefill for long prompts is slow because every prompt token runs through every layer. Earlier methods that skip late layers either require training or lose quality once more than half the layers are skipped.
Approach
RidgeFill runs layers up to a split exactly. From the split layer's hidden state, a per-layer ridge-regression map, fitted in closed form on calibration text, predicts each later layer's input. Late layers then compute only their KV cache or recurrent-state update for filled rows. The last few thousand tokens, selected rows, and all decoding steps run the full model. Evaluation uses filled-region needle retrieval, key-token loss, and splice tests, and some supporting statements are proved in Lean 4 with Mathlib.
Results
On Qwen3.8-27B in llama.cpp, time to first token is 1.25–1.31x faster at 16K–120K tokens, with RULER and LongBench v2 changes of about -1.2 to -1.4 points. On Qwen3.8-Flash-Next in vLLM, prefill is 1.50–1.60x faster, with RULER needle tasks at 99.8 versus 100.0. On the dense reference models, the fitted map retrieves 39.7 to 49.6 more filled-region points than the identity fill.
| Depth | S | RidgeFill | Identity | No fill |
|---|
| 1/4 | 8 | +0.804 | +0.409 | +1.983 |
| 3/8 | 12 | +1.006 | +0.505 | +1.591 |
| 1/2 | 16 | +0.085 | +0.563 | +1.471 |
| 5/8 | 20 | +0.030 | +0.406 | +1.225 |
Novel-token ΔNLL vs exact on Llama-3.1-8B by split depth (nats/token)