In The Evolution of Agent Memory I argued that once an LLM becomes the brain of an agent, context management becomes unavoidable: interactions pile up, the window eventually fills, and something has to go. Today’s mainstream answers are two — truncate the oldest text and re-prefill what’s kept, or selectively evict tokens by importance.
This post is about a third path, and it’s training-free: take the KV cache you prefilled over the full context, slice it down to a subset, and use that — it is often more accurate than recomputing from the same kept text. Note that the two arms keep the exact same text; the only difference is where the KV came from.
Why? Because the two objects are not the same thing. The recomputed cache has only ever seen the starved subset. The sliced cache was computed while the full context was present — so it carries a trace of the very content you deleted.
To be clear up front: that a cache can be ≥ recomputation is not itself news — “a smaller cache can beat the full one” is already documented in the eviction literature. But the field’s recipe for turning that into a quality gain is almost always to add training: learn a compressor, learn retention gates, learn which heads are retrieval heads. My claim is the opposite — the leverage was never in the training. What’s missing are three things: the variable that governs it, the causal mechanism behind it, and the free policies the mechanism hands you.
1. The governing variable: coverage
“Cache beats fresh” is conditional, and the condition is coverage — how much of the original context you keep. Sweep coverage and plot the accuracy gap reuse − fresh:
Three datasets, two model families (Qwen and Mistral), one shape: a cliff at low coverage (cache is worse), a broad bonus region in the middle (cache is better), and exactly zero at 100%.
That zero is not a coincidence — it’s the identity gate: at full coverage nothing is dropped, the two arms are the same computation, and the gap must be zero. It is, to the last item. This property is what separates a mechanism from an implementation bug: every difference appears only when you delete something, and vanishes exactly when you delete nothing.
In between, two forces fight over one knob:
- The memory bonus (+). The kept KV rows were computed with the full context present; fresh only sees the starved subset. Split items by whether the answer-evidence paragraph survived the cut, and the bonus lands exactly where theory predicts — in the evidence-dropped stratum, where the cache is the only place any trace of the evidence still lives.
- The degeneration cost (−). Push coverage too low and the cache becomes a degenerate prefix: it repeats, refuses to stop, answers off-topic. This cost concentrates in the “evidence kept, everything else gone” low-coverage stratum — precisely where fresh recomputation still has all it needs.
Below the crossover the cost wins; above it the bonus wins; at 100% both vanish. Video has no cliff at all (frames are highly redundant — it never starves), which independently confirms that the cliff is information starvation, not some position-encoding quirk.
The practical reading is already useful: don’t over-evict (the cliff is real, and it sits right where the evidence survived but little else did); and in the broad middle, the reused cache is not a degraded approximation of recomputation — it is simply better, for free.
2. What does it actually remember?
The coverage curve says the cache knows something fresh doesn’t. The cleanest way to show what is to delete the evidence outright and see who can still answer.
Text version. Multi-hop QA; physically remove the paragraph that holds the answer. Fresh has never seen the evidence; reuse holds only its trace. Reuse still reconstructs the full multi-hop chain, landing on the exact deleted answer. A real example (the paragraph containing “11 February 1929” was removed):
fresh: “The question appears to be based on a misunderstanding… there is no direct connection…”
reuse: “The author of Princeps Pastorum is Pope John XXIII, who died in Vatican City. Vatican City became an independent country on 11 February 1929, when the Lateran Treaty was signed…”
Cross-modal version, even sharper: prebake a video with its audio track, then physically delete every audio token’s KV and keep only the video rows, and ask audio-dependent questions. The audio-prebaked video cache significantly beats an identically-positioned audio-free prebake — the only difference being whether audio was present at prebake. The video tokens’ KV absorbed it.
3. The mechanism: a downstream-attention trace
So where does the information physically live?
Hypothesis. During the full prefill, attention is causal: paragraphs positioned after the evidence attended to it, so their K/V states are functions of it. Delete the evidence rows and that derived information survives in the rows you kept; fresh never had it. If true, recovery should be controlled by a purely positional quantity: how many kept tokens sit downstream of the evidence at prebake time.
Causal test (this is the key part — not correlation): same question, same kept set, and we move only the evidence paragraph’s prebake position before deleting it. gold-first = every kept paragraph attends to it (maximal trace); gold-last = nothing can (zero trace, by the causal mask). The result is clean: maximize the trace and recovery is largest (+2.4 to +5.8 pp, all five dataset × family cells significant); zero it out and recovery collapses. Put differently — remove the trace and reuse and recompute become the same model.
Ruling out lost-in-the-middle. One might suspect gold-first enjoys a primacy / attention-sink advantage. So place the evidence at a random interior slot — as long as 7–8 paragraphs sit downstream, recovery is as strong as gold-first (+4.9 pp). The predictor is downstream mass, not the privileged first position.
Content, or just a prior? The sharpest objection: maybe the trace merely nudges the model toward facts it already knows (everyone knows Vatican City, 1929). Counterfactual probe: before prebake, rewrite the answer year to a fabricated value that appears nowhere else, then delete the paragraph and ask. The cache reproduces the fabricated year where fresh never does (6:0, p = .031), while the true-year channel shows no significant difference. The prior story loses; the content story wins.
The bandwidth is low, though — roughly 1–2% verbatim per readable item. It stores gist and entity bindings, not a verbatim transcript (often the year is right but the day/month is wrong). One generation even cites a paragraph that no longer exists: “this is supported by the text provided.”
4. For agents: the rolling context
This is the deployment scenario I care about most, and it loops right back to the memory post. An agent’s context is rolling: turns accumulate, the window fills, something must go. The results above reorder today’s options:
Drop the oldest — and slice the cache, don’t recompute.
keep-lateis “truncate the oldest turns” — except you keep the surviving KV rows instead of recomputing them from the surviving text. Those rows were baked while the dropped turns were present, so they carry the absorbed trace; re-prefilling from text is the one way to actually lose it. The instinct “context changed, recompute to be safe” picks the arm that is both slower and worse.Selective mid-context eviction needs a query that doesn’t exist yet. Importance scoring (H2O / SnapKV) is query-driven, but in an agent loop the future queries are unknown. With no query, recency (keep-later) is the only signal available — and it happens to be the trace-optimal one. (This also gives StreamingLLM’s sliding window a mechanistic justification: the window isn’t just convenient, it’s the trace-richest subset shape.)
Summarize-then-evict is an engineered trace. Generate the summary while the old turns are still present, append it, then drop them — the summary tokens’ KV absorbs the full context at bake time, like any downstream token. The natural trace is free but low-bandwidth (~1–2% verbatim); the summary is a high-bandwidth trace you pay decode tokens for. And the bandwidth measurement tells you what to summarize: write what the free channel drops (verbatim detail, long chains), not what it already keeps (gist).
One honest gap: every experiment here is a single bake and a single slice. A rolling context is bake → evict → extend → evict again, and whether the trace compounds or decays across cycles is unmeasured — that’s the experiment I run next.
5. Boundaries (some honesty)
A mechanism is only useful with its sign conditions, and it doesn’t win everywhere. On HotpotQA — 2-hop, shortcut-prone, with strong distractors kept — the cache can lose a few points at low coverage, because the trace amplifies whatever was salient at prebake, including a kept, topically-adjacent distractor.
The unifying rule: the trace amplifies what was salient in the full context. It helps when the dropped content is the signal you need and the task resists shortcuts; it hurts when strong distractors are kept, coverage is low, and the task is shortcut-prone. The sign is (need for the dropped evidence) × (distractor adjacency) — not a hop-count story.
And a hard boundary: all of this requires a unified-context prebake. In ordinary corpus-RAG, chunks are baked independently — no full-context prefill, no trace, no asymmetry — so this makes no accuracy claim for generic retrieval RAG. But the qualifying regime is common: multi-turn QA over one long document, streaming eviction, and agent loops that repeatedly subset one long history.
Closing
A KV cache sliced from a full-context prefill is not a degraded approximation of recomputation; it’s a different object — one that remembers part of what you deleted. The quality is governed by coverage, explained by a causally-established downstream-attention trace, bounded by salience amplification, and exploitable by training-free policies.
One sentence for engineers: when the kept set changes, slicing the old cache is both cheaper and more accurate than re-prefilling — except below the coverage cliff, which is detectable at runtime.
The full paper (a preprint typeset in the ICLR template): download the PDF. All experiment code: github.com/oklen/sprag.