wtf is a sliding-window kv cache? (and what it costs you)

Say you’re running gpt-oss-20b in llama.cpp on a 16 GB RTX 5070 Ti, and you want its full 131,072-token context. The weights take 11.27 GiB. In my Haiku post I worked out that this model’s KV cache (the memory it keeps for every token in the conversation) needs 48 KiB per token. So the full context needs another 6 GiB, and 17.3 GiB doesn’t go into a 16 GB card. But llama-server only asks for about 3 GiB of cache, and the model fits. The missing half comes from a detail I only gave one sentence last time: half of this model’s layers only ever look at the last 128 tokens.

A sliding-window KV cache is the engine keeping only those last few tokens for the layers that can’t see any further back. It’s a real saving, and llama.cpp and vLLM both do it by default as of October 2026. It also has a cost that shows up later, when you edit an earlier message and the server rereads the whole conversation. That’s why llama.cpp still ships a flag called --swa-full that turns the saving off, and why I think most people running this model locally should know when to use it.

first, what’s a kv cache again?#

A model writes its answer one token at a time. For each new token, every attention layer (the part of the model that looks back at earlier tokens) compares it with a key and a value saved for every earlier token. Those saved keys and values are the KV cache. I did the math for Llama 3.1 8B in serving one model to 64 people: 128 KiB per token, with a few long-prompt users holding most of the GPU’s memory.

The cache also saves work between requests. Reading a prompt and building its cache is called prefill. Often a new request starts with the same tokens as an earlier one, like the same chat history plus one new message. The engine reuses the cache for that shared start and only prefills the new part. That’s prefix caching, and it’s the thing a sliding window makes harder.

what does a sliding window change?#

Llama 3.1 8B uses full attention in every layer, so every token can look at every earlier token. Sliding-window attention (SWA) limits a layer to the last N tokens. gpt-oss-20b mixes the two. Its config.json lists 24 layers in layer_types, alternating sliding_attention and full_attention, with these sizes:

{
  "num_hidden_layers": 24,
  "num_key_value_heads": 8,
  "head_dim": 64,
  "sliding_window": 128
}

So a sliding layer working on token 20,000 only reads the 128 most recent tokens. It will never read token 5,000 again, so there’s no reason to keep token 5,000’s entry for that layer. The 12 full layers still need every token. The window saves compute too, since each sliding layer compares a new token with at most 128 others instead of the whole conversation.

Older tokens don’t stop mattering, though. The full layers still read all of them, and each layer hands what it found to the next one. The Mistral 7B paper, which used a 4,096-token window in every layer, puts it this way:

Note that tokens outside the sliding window still influence next word prediction.

If you’ve set up log rotation, it’s the same idea. A service that only ever needs the last hour of logs can delete everything older, and disk use stops growing. Where it breaks is the part that matters later. Deleting an old log line doesn’t change the lines after it, but a token dropped from a sliding layer shaped everything the model computed after it. The engine can’t rebuild that entry from the entries it kept. Getting it back means running the model over the conversation again.

how much memory does it actually save?#

Every layer stores a key and a value for each of the 8 KV heads, 64 numbers each, in a 2-byte format (llama.cpp uses F16 by default):

per layer, per token:
  2 (K and V) * 8 KV heads * 64 dims * 2 bytes = 2 KiB

full cache:     24 layers * 2 KiB * context
sliding window: 12 layers * 2 KiB * context
              + 12 layers * 2 KiB * sliding cells

llama.cpp doesn’t keep exactly 128 tokens for the sliding layers. In src/llama-kv-cache-iswa.cpp the sliding cache is the window plus room for one batch of new tokens (-ub, 512 by default), rounded up to a multiple of 256. If several slots share one cache, the window counts once per slot. With one slot it’s 128 + 512 = 640, rounded up to 768 cells, or 18 MiB across 12 layers. At a few context sizes with -np 1:

context (-c)	full cache	sliding-window cache	saving
4,096	192 MiB	114 MiB	1.68x
32,768	1,536 MiB	786 MiB	1.95x
131,072	6,144 MiB	3,090 MiB	1.99x

Start llama-server with -c 32768 -np 1 and the startup log lists two caches: a non-SWA one of 32,768 cells (768 MiB) and an SWA one of 768 cells (18 MiB). The last row is the 5070 Ti problem from the start. 11.27 GiB of weights plus 6 GiB doesn’t fit in 16 GB. With 3.02 GiB instead it comes to about 14.3 GiB, which leaves room for compute buffers and whatever Windows is drawing on that card.

The saving stops at 2x because only half the layers slide. When every layer stores the same amount per token, the most a sliding window can save at long context is total layers divided by full-attention layers. Mistral 7B v0.1 is the other end of that. Its config.json has 32 layers, 8 KV heads of 128 numbers (4,096 / 32), and a sliding_window of 4,096 that applies to every layer. That’s 4 KiB per layer and 128 KiB per token, the same as Llama 3.1 8B. A 32,768-token context needs 4 GiB with a full cache, and 512 MiB if the cache stops at 4,096 tokens, which is 8x. That matches what the paper reports for its rolling buffer cache:

On a sequence length of 32k tokens, this reduces the cache memory usage by 8x, without impacting the model quality.

You only get that if the engine reads the field, though. llama.cpp converts MistralForCausalLM into its plain Llama architecture, which ignores sliding_window. So in llama.cpp this model gets a full cache.

so what’s the catch?#

The engine throws away sliding-layer entries as the window moves, and most of the time nobody misses them. They matter when a new request matches an old one only up to some point in the middle. You’re 30,000 tokens into a chat and you edit your third message, or regenerate an older reply, or your agent framework trims old tool output from its history. The new prompt matches the cached one up to some token and then differs.

With a full cache, the engine cuts every layer back to that token and prefills from there. With a sliding-window cache, the full layers are fine, but the sliding layers only hold the last few hundred tokens of the old conversation. They have nothing for the 128 tokens before the split, so llama.cpp falls back to rereading the prompt from token 0. The source comment calls it “forcing full prompt re-processing due to lack of cache data”.

gpt-oss-20b hits this even without edits. It’s a reasoning model, and its chat format drops the previous turn’s reasoning from the next request. So every follow-up differs from the cached prompt right after your last message. In llama.cpp issue #15082 in August 2025, slaren, who works on llama.cpp, wrote about the 120B version:

Almost every new message causes a full reprocess of the entire context, which really impacts the usability of the model.

Georgi Gerganov, who maintains llama.cpp, replied that the cause was the small 128-token window.

A reread costs prefill time, and that depends a lot on the machine. A published llama-bench run on an RTX 5070 Ti (llama.cpp build 8077, -fa 1 -ub 1024, Ubuntu 24) prefilled a 32,768-token gpt-oss-20b prompt at 7,106 tokens per second, which comes to about 4.6 seconds. The llama.cpp gpt-oss guide has a 36 GB M4 Max taking 55.7 seconds for the same 32,768 tokens (llama-batched-bench, build 6123, August 2025). Four seconds per edit is annoying, and close to a minute is what slaren meant by “impacts the usability of the model.”

what llama.cpp does about it#

All of this is as of llama.cpp’s source in October 2026.

• A small cache by default. Since the SWA PR merged in May 2025, llama.cpp keeps two caches for models like this one and prunes old tokens from the sliding one. It also turns off --context-shift and --cache-reuse for these models, with a warning, since both need to move old entries around.

• Context checkpoints. PR #15293, from August 2025, saves copies of the sliding layers’ recent entries while reading a prompt. It takes one near the end of each prompt, one at your last message, and earlier ones at least 8,192 tokens apart (--checkpoint-min-step), up to 32 per slot (--ctx-checkpoints). They live in system RAM, and the PR calls each one “relatively small (proportional to the SWA window that the model uses)”. When a prompt splits off, the server restores the newest checkpoint before the split and prefills from there. That covers normal chat, including the dropped reasoning. An edit far back still goes back to an older checkpoint, or to token 0.

• --swa-full. This makes the sliding cache as big as the full one and stops the pruning, so every reuse trick works again and checkpoints aren’t needed. The SWA PR explains it:

  Note that advanced cache operations such as removing tokens or shifting their positions are not possible when using SWA cache, because token information becomes lost when the window slides. For such cases, we can “fallback” to the old implementation by expanding the SWA cache size to the full context and disabling the SWA token pruning. This of course would lead to more memory usage.

  For gpt-oss-20b the extra memory is 24 KiB per token of context: 768 MiB at 32K, 3 GiB at 128K.

what vllm and sglang do#

vLLM got its hybrid KV cache manager in PR #17996 in June 2025. It keeps one pool of fixed-size blocks for all layers, and sliding layers give back blocks once they fall out of the window. The design doc explains why it didn’t go with a fixed ring of slots:

But this naive implementation is not compatible with prefix caching so we didn’t pick this design. In vLLM, we allocate different blocks for different tokens and free blocks that are outside the sliding window.

Freed blocks go into the same least-recently-used queue as everything else and stay findable until something else needs the memory. A prefix hit needs the full layers’ blocks for the whole shared start, plus the sliding layers’ blocks for the window before the split. So reuse works while the GPU has spare memory and gets worse when it’s busy. vLLM’s version of --swa-full is --disable-hybrid-kv-cache-manager. The warning it logs is honest about the trade: “we do not enable any optimizations for saving KV cache memory (e.g., dropping the KV cache outside the sliding window). The compute of layers like sliding window is still saved.”

Btw, this also corrects something in my Haiku post (it’s fixed now). I first wrote that vLLM’s numbers matched the full 48 KiB per token, based on an A6000 log that reported 628,512 tokens for 28.77 GiB. vLLM builds from that time (the v0.10.1 source shows it) printed that count as if every layer kept every token. The allocator underneath was already freeing sliding blocks, and current vLLM prints a count that accounts for the window.

SGLang keeps a separate pool for sliding layers, sized by --swa-full-tokens-ratio (0.8 by default, so 0.8 sliding-layer slots for every full-layer slot). For gpt-oss-20b that’s 24 KiB + 0.8 * 24 KiB = 43.2 KiB per token instead of 48, about 10% less. The extra room keeps old sliding entries around, and its prefix cache only counts a match if a full window of them before the split is still there. A lower ratio saves more memory and gets fewer hits, and --disable-hybrid-swa-memory turns the separate pool off.

when should you keep the full cache?#

My rule is to work out what a miss costs and how often you’ll take one. A miss costs your prompt length divided by your prefill speed. The full cache costs, for gpt-oss-20b, 24 KiB times your context. Then:

• Keep the small cache when memory is the limit. The 128K context on a 16 GB card only fits this way. On a shared server, the saved half is room for twice as many tokens in flight, and for append-only chat, checkpoints already handle most reuse.

• Turn on --swa-full when you branch a lot and have memory to spare. That means editing old messages, regenerating, agent frameworks that rewrite their history, or many requests sharing a long start that split at different points. On a Mac, where the guide’s M4 Max needed close to a minute for a 32K reread, I’d turn it on for any long chat that fits. On the 5070 Ti at 32K, the extra 768 MiB fits next to 11.27 GiB of weights, so I’d turn it on there too unless I need the long context.

• On vLLM, leave the hybrid manager on and watch the hit rate. Compare vllm:prefix_cache_hits with vllm:prefix_cache_queries. If hits fall off when the server gets busy, sliding blocks are being evicted before reuse. I’d add memory or cut concurrency before reaching for the flag, since the flag takes memory away from everything else.

To see which case you’re in with llama.cpp, start the server both ways and send the same conversation twice, with an edit to an early message the second time:

llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 32768 -np 1 --port 8080
llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 32768 -np 1 --port 8080 --swa-full

curl -s localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" -d @chat.json \
  | jq '.timings | {cache_n, prompt_n}'

cache_n is how many prompt tokens came from the cache, and prompt_n is how many the server had to prefill. If prompt_n jumps to the whole conversation after a small edit, you’re paying for the reread. Divide it by your prefill speed and you have the number to weigh against the memory.