does fp8 change the answers?

Say you’re serving Llama 3.1 8B Instruct from one rented GPU, like the L40S (an NVIDIA datacenter card with 48 GB of memory) I used in the 64 users post. Somebody reads that FP8 cuts the weights in half and makes tokens come out faster, and asks why you aren’t using it. So you switch, ask the model three questions, get three answers that look about the same, and ship it. A few months ago I would have done exactly that. The speed part holds up, and the math below shows why. The “looks the same” part doesn’t, because a few matching prompts say very little about whether the model still behaves like itself.

wtf is fp8?#

FP8 means storing each of the model’s numbers in 8 bits instead of 16. A number format is the rule a computer uses to pack a number into a fixed number of bits. Floating-point formats split those bits into a sign, an exponent (roughly, which power of two you’re near), and a mantissa (how finely you can place a number between two powers of two). More exponent bits buy range, and more mantissa bits buy precision.

Llama 3.1 8B ships in BF16: 1 sign bit, 8 exponent bits, and 7 mantissa bits (Google’s bfloat16 docs). The FP8 format vLLM uses for weights is called E4M3, and the name is the recipe: 4 exponent bits, 3 mantissa bits, plus the sign. Per the vLLM FP8 docs, the largest number it can hold is 448. Three mantissa bits isn’t much. Between 2 and 4, E4M3 has exactly eight values (2, 2.25, 2.5, and so on up to 3.75), so pi gets stored as 3.25. BF16 has 128 steps in the same range and stores pi as 3.140625.

The closest everyday thing is saving a photo as a high-quality JPEG. You throw away fine detail, and at normal size the picture looks the same. The analogy breaks in one place that matters. A JPEG’s error stays in its pixel. A model’s rounding error feeds into the next layer, through all 32 layers, and then into the choice of the next token. That token becomes input for every token after it, which is what the second half of this post is about.

what is the scale factor for?#

Model weights are mostly small numbers, and E4M3 is bad at small numbers. Near zero its smallest step is 0.00195, so anything under about 0.001 rounds to zero. The fix is a scale factor. Before converting, divide a block of weights by one number so the biggest weight lands at 448. Then store the FP8 values plus that one scale, and multiply back when you use them.

Say the biggest weight in a matrix is 0.05. The scale is 0.05 / 448 = 0.000112. A weight of 0.001 becomes 8.96, rounds to 9, and comes back as 0.001004. Without the scale it would come back as 0.00195, almost double, and a weight of 0.0004 would become zero. The main choice is how many weights share one scale:

Per-tensor uses one scale for a whole weight matrix. It’s simple, but one unusually big weight stretches the scale for every other number in the matrix.
Per-channel uses one scale per output row, so a big value in one row doesn’t cost the other 4,095 rows their precision.

The same vLLM page, which covers making FP8 checkpoints with llm-compressor, recommends per-channel weights. It also converts the activations (the numbers flowing between layers while the model runs) to FP8, with a fresh scale computed for each token. That’s what W8A8 means: 8-bit weights and 8-bit activations. Red Hat publishes both styles for this model:

checkpoint	weight scales	activation scales	size
`meta-llama/Llama-3.1-8B-Instruct`	none (BF16)	none (BF16)	16.06 GB
`RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8`	per-tensor	per-tensor, calibrated on 512 UltraChat samples	9.08 GB
`RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`	per-channel	per-token, computed at run time	9.08 GB

Both keep the embeddings and lm_head (the last matrix, which scores every word in the vocabulary) in BF16, which is why they’re 9.08 GB and not 8. You can also skip the download and let vLLM convert the BF16 weights while loading. Btw, the flag for that changed: as of vLLM 0.31.0 (October 5, 2026), --quantization fp8 on a BF16 checkpoint errors out. The new names in the online quantization docs are fp8_per_tensor and fp8_per_channel:

# convert at load time, one scale per output row
vllm serve meta-llama/Llama-3.1-8B-Instruct --quantization fp8_per_channel

# or load a checkpoint that's already FP8
vllm serve RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic

vLLM does the FP8 math natively on NVIDIA GPUs with compute capability (NVIDIA’s version number for a chip’s features) 8.9 or newer: Ada (the L40S and the RTX 40 series), Hopper, and Blackwell (my 5070 Ti). Older cards like the A100 still load FP8 checkpoints, but they only store the weights in FP8 and do the math in 16 bits. On Windows, vLLM needs WSL, the Windows Subsystem for Linux (vLLM GPU install docs).

why should half the bytes mean faster tokens?#

I covered this in the V-cache post: with one user, writing each token (decode) means reading every weight from GPU memory once, and the math is small next to the reading. So the speed limit is memory bandwidth divided by bytes read per token. From the model’s config.json, the linear layers hold 6.98B parameters and lm_head holds 0.53B. Decode reads all of those, plus one row of the embeddings:

BF16: (6.98B + 0.53B) * 2 bytes            = 15.0 GB per token
FP8:   6.98B * 1 byte + 0.53B * 2 bytes    =  8.0 GB per token

L40S at 864 GB/s:  864 / 15.0 =  57.6 tok/s ceiling (BF16)
                   864 /  8.0 = 107.6 tok/s ceiling (FP8)

That’s 1.87x fewer bytes, not 2x, because lm_head stays in BF16. The KV cache (the saved state for earlier tokens) adds about 0.15 GB per token at a 1,000 token prompt, around 1%, so I left it out. The 864 GB/s is from NVIDIA’s L40S page.

The BF16 run in the 64 users post did 46 tok/s for one user, which is 80% of the 57.6 ceiling. At the same 80%, FP8 comes to roughly 86 tok/s. It may land lower, since converting activations to FP8 adds a little work at every layer, and the vLLM docs say latency gains are limited in the per-tensor online mode. Their “up to a 1.6x improvement in throughput” doesn’t name the GPU or the load, and throughput is a many-users number anyway.

The second win matters more on this card, I think. FP8 frees about 7 GB, and at 128 KiB of KV cache per token that’s room for about 53,000 more tokens. KV cache is what ran out when the long-document users showed up in the 64 users post. Reading prompts (prefill) is limited by math rather than memory. Per that same NVIDIA page, the L40S does FP8 math at twice its BF16 rate, so long prompts should speed up too.

On the 5070 Ti it’s a different question. The BF16 weights are 14.96 GiB on a 16 GB card, which leaves almost nothing for the KV cache, while FP8 is 8.46 GiB. At 896 GB/s the FP8 ceiling is 111.6 tok/s, and the same 80% works out to about 89 tok/s. On that card, FP8 is what makes this model usable in vLLM at all.

is it still the same model?#

The published numbers say it’s very close. Red Hat’s model cards compare both FP8 checkpoints with BF16, using vLLM and their fork of lm-evaluation-harness with Meta’s prompt formats where they exist:

benchmark	BF16	FP8 per-tensor	FP8-dynamic
MMLU, 5-shot	67.95	67.97	68.02
ARC Challenge, 0-shot	82.00	81.66	81.2
GSM8K CoT, 8-shot	81.96	81.12	82.0
Winogrande, 5-shot	78.45	77.90	77.7
OpenLLM v1 average	73.79	73.44	73.6

Sources: the per-tensor card and the dynamic card, which rounds to one decimal. MMLU and ARC are multiple-choice tests, and GSM8K is grade-school math word problems. The biggest drop is 0.84 points on GSM8K. With 1,319 questions at about 82%, one standard error is sqrt(0.82 * 0.18 / 1319) = 1.06 points, so that drop is inside the noise. Red Hat’s paper “Give Me BF16 or Give Me Death” tested the whole Llama 3.1 family and calls FP8 “effectively lossless across all model scales.”

I believe those tables. They answer a narrower question than mine, though. An accuracy score counts how many answers were right, not whether they were the same answers. The paper “Accuracy is Not All You Need” from Microsoft Research calls the difference flips: answers that go from right to wrong, or wrong to right, after compression. The two directions roughly cancel, so accuracy stays flat while answers change underneath.

Their example is Llama 2 70B with bitsandbytes 8-bit quantization: 0.55% away from the original on MMLU, with 4.6% of answers flipped. FP8 wasn’t one of the formats they tested, so this says nothing bad about FP8 by itself. It does mean an accuracy table is the wrong place to look for flips.

why three matching prompts prove almost nothing#

At temperature 0, the model always picks its top-scoring token (greedy decoding). For most tokens the top choice wins by a mile, and FP8’s rounding can’t change that. It only matters when the top two are nearly tied. Then a tiny nudge flips the pick, and from there the two models continue different text. That’s the snowball: one flipped token, and everything after it builds on a different prefix.

llama.cpp publishes how often this happens. Its perplexity README compares Llama 3 8B in BF16 and FP16, two 16-bit formats, on the Wikitext-2 test set. They pick the same top token 99.739% of the time, so about 1 position in 380 differs between two formats nobody calls lossy. At that rate, a 256-token answer has a 1 - 0.99739^256 = 49% chance of at least one flip. Wikitext is harder to continue than a chat answer, so the real rate in chat is lower, but different and equally good text is a normal outcome even without quantization.

Red Hat’s paper measured this on Arena-Hard prompts with greedy decoding. For the 8B quantized models, ROUGE-L against the BF16 output (roughly, how much of the word order matches) dropped as low as 0.46, while BERTScore (similar meaning, not similar words) stayed around 0.92. So the wording moves a lot and the meaning mostly doesn’t. Matching prompts mean you picked prompts with no near ties. A mismatch doesn’t mean anything broke either. vLLM also doesn’t promise identical output for the same model by default, because batching changes the order of the math (vLLM reproducibility docs).

check one: compare probabilities on the same text#

Feed both models the exact same text and compare the probabilities each one gives the next token at every position. Neither model writes anything, so nothing can snowball. The standard number is KL divergence (Kullback-Leibler): at each position, how far the FP8 model’s probabilities over the whole vocabulary are from the BF16 model’s, averaged over the text. Zero means identical.

llama.cpp has the reference tool. You save the full-precision model’s logits (raw scores) once, then score the quantized model against them:

./scripts/get-wikitext-2.sh
llama-perplexity -m Llama-3.1-8B-Instruct-BF16.gguf \
  -f wikitext-2-raw/wiki.test.raw --kl-divergence-base llama31-bf16.kld
llama-perplexity -m Llama-3.1-8B-Instruct-Q8_0.gguf \
  -f wikitext-2-raw/wiki.test.raw --kl-divergence-base llama31-bf16.kld \
  --kl-divergence

It prints mean KL divergence, “Same top p” (how often both pick the same top token), and RMS Δp (the root mean square of how much the correct token’s probability moves). For Llama 3 8B, the README lists Q8_0 at a mean KL divergence of 0.001355. The two models also pick the same top token 97.674% of the time. llama.cpp has no FP8 type, so it can’t load the Red Hat checkpoints. Q8_0 is a different 8-bit format (integers with one scale per 32 weights), and it’s the closest published reference point I found.

In vLLM, prompt_logprobs gives you scores for every position of a prompt. It returns the top 20 tokens plus the real next token, not all 128,256, so exact KL divergence is out of reach. The other llama.cpp numbers are exact, though. Run this once per model:

# dump_logprobs.py MODEL OUT.json [QUANTIZATION]
import json, sys
from vllm import LLM, SamplingParams

model, out_path = sys.argv[1], sys.argv[2]
quantization = sys.argv[3] if len(sys.argv) > 3 else None
llm = LLM(model=model, quantization=quantization, max_model_len=4096)

token_ids = llm.get_tokenizer().encode(open("wikitext-2-raw/wiki.test.raw").read())
chunks = [token_ids[i:i + 2048] for i in range(0, 2048 * 100, 2048)]
params = SamplingParams(max_tokens=1, prompt_logprobs=20)
outputs = llm.generate([{"prompt_token_ids": c} for c in chunks], params)

rows = []
for chunk, output in zip(chunks, outputs):
    for pos in range(1, len(chunk)):
        scores = output.prompt_logprobs[pos]
        top_token = min(scores, key=lambda t: scores[t].rank)
        rows.append({"real": scores[chunk[pos]].logprob, "top": top_token})
json.dump(rows, open(out_path, "w"))

# compare_logprobs.py BF16.json FP8.json
import json, math, sys

base, quant = json.load(open(sys.argv[1])), json.load(open(sys.argv[2]))
pairs = list(zip(base, quant))
same_top = sum(b["top"] == q["top"] for b, q in pairs) / len(pairs)
delta_p = [math.exp(q["real"]) - math.exp(b["real"]) for b, q in pairs]
rms_delta_p = math.sqrt(sum(d * d for d in delta_p) / len(delta_p))
ppl_ratio = math.exp(sum(b["real"] - q["real"] for b, q in pairs) / len(pairs))
print(f"same top token {same_top:.3%}, RMS dp {rms_delta_p:.3%}, PPL ratio {ppl_ratio:.4f}")

Dump meta-llama/Llama-3.1-8B-Instruct and RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic, then compare them. Do the BF16 dump twice and compare those as well, so you know the noise floor. Wikitext is fine for a first pass, but text that looks like your real traffic is better.

check two: let both models write and count flips#

This one matches how people use the model. Both models answer the same questions freely, and you compare answer by answer, not only the final score. lm-evaluation-harness can drive vLLM, and --log_samples saves every answer:

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3.1-8B-Instruct,add_bos_token=True,max_model_len=4096 \
  --tasks gsm8k --num_fewshot 5 --batch_size auto \
  --output_path results --log_samples

Run it again with pretrained=RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic. Keep add_bos_token=True: the vLLM docs warn that quantized models can be sensitive to a missing start token, and lm-eval doesn’t add it by default. Each run writes a samples_gsm8k_<date>.jsonl file, and counting flips takes a few lines:

# flips.py BF16_SAMPLES.jsonl FP8_SAMPLES.jsonl
import json, sys

def correct_by_question(path):
    rows = [json.loads(line) for line in open(path)]
    return {r["doc_id"]: r["exact_match"] for r in rows if r["filter"] == "strict-match"}

base, quant = correct_by_question(sys.argv[1]), correct_by_question(sys.argv[2])
flips = [q for q in base if base[q] != quant[q]]
print(f"{len(flips)} of {len(base)} answers flipped ({len(flips) / len(base):.1%})")

GSM8K works well here. Every answer needs a few hundred tokens of reasoning, so the snowball has room to happen, and the final answer is a number, so grading is exact. If you have your own prompts with checkable answers, use those as well.

what i’d do before switching#

FP8 on this model looks like the safest quantization there is, and the cards back that up. I’d still run it against my own traffic, in this order:

Measure the noise floor. Run BF16 against BF16 for both checks. Every FP8 number only means something next to that one.
Compare probabilities. If FP8’s same-top-token rate and RMS Δp sit near the BF16 noise floor, and well inside the Q8_0 reference from llama.cpp, move on.
Count flips. My rule: if accuracy is within one standard error (about 1 point on GSM8K) and flips are no more than about twice the BF16-vs-BF16 count, I’d ship it. If it fails, try the per-channel checkpoint before giving up on FP8.
Measure speed last. Run vllm bench latency --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic --input-len 1000 --output-len 256 --batch-size 1 and the same for BF16, and see how close FP8 gets to the roughly 86 tok/s the bandwidth math allows.