Argues that the industry missed this shift because pre-quantization CPU inference was compute-bound and hopeless, but modern Q4 quantization changes the math entirely. Their T5600 hits 5 tok/s not because the Xeon is fast, but because DDR3-1600 across two sockets delivers ~85 GB/s — enough to stream a 15 GB Q4 model at usable speeds.
Explicitly distinguishes this result from the usual 0.4 tok/s 'technically works' demos. At 5 tok/s a streamed response is fast enough for single-user chat or async agent loops, which reframes what counts as 'production-capable' hardware.
A $180 eBay workstation drawing 220W costs about $0.03/hour to run versus $1.00/hour for an AWS A10G. Even though the A10G is ~9x faster, the 33x cost gap means the Xeon wins decisively for anything that doesn't need bursty throughput.
The 5 tok/s result depends on NUMA-aware thread pinning, disabled turbo boost for clock stability, mlock'd weights, and manually forced 16 threads per socket. Default llama.cpp settings would leave significant performance on the table, suggesting most people benchmarking CPU inference are measuring their configuration mistakes.
NeoMind Labs published a teardown on running Google's Gemma 4 26B at a sustained 5 tokens/second on a Dell Precision T5600 — a workstation that shipped in 2013 with a pair of Xeon E5-2670 CPUs, 128 GB of DDR3-1600 ECC, and no discrete GPU. The whole rig, sourced off eBay, cost roughly $180. Inference runs entirely on CPU via a patched build of `llama.cpp` with Q4_K_M quantization, bringing the 26B model down to about 15 GB on disk and roughly 17 GB resident.
The numbers matter because they're not the usual "look, it technically works" 0.4 tok/s demo. Five tokens per second is roughly the threshold where a streamed response becomes usable for a single-user chat or an async agent loop — slower than a GPU, faster than most people expected from a decade-old CPU. The post walks through the actual tuning: NUMA-aware thread pinning (`--numa distribute`), disabling turbo boost to keep clocks stable, mlock'ing the weights so the kernel doesn't page them, and forcing 16 threads per socket instead of letting llama.cpp guess.
For context: an A10G on AWS runs the same model at roughly 45 tok/s and costs about $1.00/hour on-demand. The Xeon rig draws around 220W under load, which at US grid prices is closer to $0.03/hour.
The reflexive read is "cool hack, doesn't scale." That's wrong, and it's wrong in an interesting way. The bottleneck for CPU inference on modern quantized models isn't FLOPS — it's memory bandwidth, and DDR3 on a dual-socket board from 2013 delivers around 85 GB/s aggregate, which is only ~3x slower than a single-channel DDR5 desktop. A 15 GB Q4 model at 5 tok/s needs roughly 75 GB/s of sustained read throughput. The Xeon has it. The reason nobody noticed is that until quantization got good enough to fit a useful model in RAM, CPU inference was compute-bound and hopeless.
This reframes a bunch of assumptions the industry has been operating on since 2023. The narrative that you need H100s (or at minimum a 4090) to self-host anything past 13B parameters was true for FP16. It stopped being true somewhere around the release of `k-quants` in llama.cpp, and the gap has been widening quietly ever since. Gemma 4's architectural tweaks — grouped-query attention with a KV cache small enough to not blow out the L3 — happen to be almost perfectly suited to the CPU case, whether Google intended that or not.
The HN thread on the post is worth reading for the counter-arguments. The strongest one: 5 tok/s is fine for one user, but throughput per dollar on a shared GPU is still dramatically better once you have concurrent requests. That's true. It's also irrelevant to most self-hosting scenarios, which have exactly one user (you), or to agent workflows where latency matters more than QPS. The GPU-per-user math has been dragging small teams into managed inference contracts they don't actually need, and this post is a data point that the alternative is boring old hardware you can buy for the price of a nice dinner.
Worth naming what this isn't: it's not a path to serving production traffic. It's not going to run Llama 4 405B. It doesn't help you if your workload is prefill-heavy (long documents, RAG with big contexts) because the prefill phase is compute-bound and the Xeon eats it alive there. The post is honest about all of this — the author explicitly says prefill on a 4K context takes 40+ seconds, which makes this a bad fit for anything document-heavy.
If you're running a personal assistant, a coding agent that ticks over in the background, or an internal tool at a company small enough that "one concurrent user" is a reasonable ceiling, the economics just shifted. A used Xeon workstation with 128 GB of RAM is roughly two months of a modest GPU rental. It sits under a desk. It doesn't have an autoscaler that wakes up at 3am to bill you $40 because Kubernetes decided to reschedule a pod.
The practical checklist if you want to try this:
- Any dual-socket Xeon E5 v1/v2 with 8+ channels of DDR3 will get you in the ballpark. E5-2670, E5-2680 v2, E5-2690 v2 are the sweet spots on the used market. - You need enough RAM to hold the quantized model plus KV cache plus OS — 64 GB is tight for 26B, 128 GB is comfortable, 256 GB opens the door to 70B models at Q4. - Use `llama.cpp` from source with `LLAMA_BLAS=1` and OpenBLAS. Don't bother with AVX-512 tuning on Sandy Bridge; you don't have it. AVX2 is where the wins are. - NUMA is the whole game on dual-socket: `--numa distribute` with `--threads` set to physical cores per socket, not total cores. Hyperthreading hurts here.
For teams: this is a legitimate answer to the "can we self-host our internal Slack bot / code review assistant / docs Q&A" question that used to require a GPU budget conversation. It's not the answer for user-facing product inference. Know which one you're building.
The interesting second-order effect is what this does to the used enterprise hardware market. Retired Xeon and EPYC servers have been trading at scrap-metal prices because nobody wanted them for anything. If the "one workstation, one model" pattern catches on with the self-hosting crowd, expect that curve to bend. More importantly, expect the next generation of open-weight models to get architected with this constraint in mind — Gemma 4's KV cache design already reads like someone at Google was thinking about it. The GPU cartel isn't going anywhere for training, but for inference at the edge of the long tail, the boring hardware is quietly catching up.
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an in
A dual Xeon of this era is probably pulling 300W or more when loaded.At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows)https://news.ycombinator.com/item?id=48354801
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chatThis is a GPT4 level model running local