Topic

hardware

3 public notes.

TechAug 31, 20266 min read

What a power limit should do to a 5070 Ti running llama.cpp

Generation mostly waits on memory, so a lower power limit should barely slow it down.

aillminferencehardware

TechAug 14, 20266 min read

When the model doesn't fit in 16 GB

A 32B model split between GPU and RAM runs at 9 tokens per second. A 120B MoE model runs at 21.

llminferencehardwaregpu

TechJul 30, 20266 min read

Does 96 MB of V-cache help an LLM?

The 9800X3D has a huge L3 cache. CPU inference mostly runs into the memory link anyway.

llminferencehardwarebenchmark