The drafter-verifier split is a new inference architecture, not a hack

What happened

A blog post titled *"A 10 year old Xeon is all you need (for 26B-A4B MTP Drafters without GPU)"* hit 682 points on Hacker News, and the comments aren't the usual hardware-fetish theater. They're a working group. People are swapping motherboard model numbers, DDR4 timings, NUMA-pinning incantations, and llama.cpp branches. The headline number — a $40 Xeon E5-2680 v4 from 2016 holding ~8 tok/sec on Gemma 3 27B without touching a GPU — is the hook. The actual finding is structural.

The technique is Multi-Token Prediction (MTP) speculative decoding. A small, fast "drafter" model (here a 4B-parameter model proposing tokens for a 26B target) runs ahead, and a larger "verifier" accepts or rejects its proposals in one batched pass. Speculative decoding isn't new — DeepMind published the core idea in 2022, and llama.cpp shipped a usable implementation in 2023. What's new is the realization that the drafter and verifier are two completely different computational workloads that want opposite hardware.

The drafter is autoregressive, single-batch, and memory-bandwidth-bound: it streams a few billion weights from RAM per token and does almost no arithmetic. The verifier does a single forward pass over a draft of N tokens — compute-heavy, batch-friendly, the thing GPUs were built for. Run them on the same accelerator and you waste the GPU's compute during the drafter's serial slog. Split them across two devices optimized for what they actually do, and the math changes.

Why it matters

The 2016 Xeon E5-2680 v4 isn't special because it's old. It's special because server boards from that era ship quad-channel DDR4. Four channels of 2400 MT/s DDR4 yield 76.8 GB/s of memory bandwidth. A modern consumer desktop with dual-channel DDR5-6000 lands around 96 GB/s on paper but routinely measures 50–60 GB/s under real load, with worse latency under the kind of strided access patterns a transformer's KV cache produces. For a memory-bound 4B drafter, the decade-old server CPU genuinely wins.

This inverts the usual hardware narrative. The reason nobody noticed until now is that nobody was running this workload until MTP made it useful. Single-model CPU inference on a 27B model is slow because you're paying memory-bandwidth tax on 27B weights per token. With MTP, you pay it on 4B weights per *proposed* token, and the GPU pays compute tax on N tokens at once during verification. The bottleneck shifts, and the optimal hardware shifts with it.

The HN thread surfaces a deeper community pattern. The top-voted comments aren't "can it run Crysis" jokes — they're procurement intel: which Supermicro X10DRi boards have working IPMI, which ECC RDIMM kits don't have SPD bugs, which BIOS revisions unlock all four memory channels properly. This is the homelab community pivoting from "my GPU is too small" to "my memory bus is too narrow," and that's a meaningful shift in how practitioners think about inference cost.

Compare this to the dominant inference-optimization narrative of 2024–2025: quantization (GGUF, AWQ, GPTQ), kernel fusion (FlashAttention), and KV-cache compression. All of those squeeze more from a single accelerator. MTP-with-CPU-drafter does something different — it admits that inference is not one workload and partitions it across devices that are each cheap at their job. The drafter is $40 of e-waste. The verifier is the H100 you already have. The total cost-per-token curve bends.

What this means for your stack

If you're running production inference, the immediate action is to measure what fraction of your GPU time is the drafter path on whatever speculative-decoding setup you have. vLLM and TensorRT-LLM both support speculative decoding; neither defaults to splitting drafter and verifier across devices. If your drafter is sitting on the same A100/H100 as your verifier, you're paying ~$3/hr to run a workload that an old Xeon would do for power costs alone. The orchestration story isn't solved yet — you need to think about PCIe transfer overhead for the drafted token sequences and about pipelining so the verifier doesn't stall — but the cost asymmetry is large enough that someone is going to ship a clean split-deployment runtime within six months.

For homelabbers and indie devs, the calculus is simpler: a used dual-socket E5-2680 v4 server with 128GB of DDR4 ECC costs less than a single RTX 4070. You don't need a GPU to run usable LLMs anymore — you need memory bandwidth and patience, and one of those is now extremely cheap. The 8 tok/sec figure is read-aloud speed; not interactive coding-assistant fast, but absolutely fine for batch summarization, overnight document processing, or backend agent workflows where latency tolerance is measured in seconds.

For cloud providers, this is the start of an awkward conversation. AWS, GCP, and Azure all price inference by GPU-hour. They don't have a SKU for "big DDR4 box with no accelerator," because until ~6 months ago there was no reason to want one. Expect bare-metal providers (Hetzner, OVH, Vultr) to capture this workload first — their inventory already includes the old Xeon hardware that hyperscalers retired.

Looking ahead

The deeper story is that LLM inference is starting to look less like "a model running on a GPU" and more like a multi-phase pipeline that gets optimized one phase at a time, each on its own hardware. The drafter-verifier split is the first crack. The next will probably be embedding generation, KV-cache management, and tool-call routing all peeling off onto specialized substrates. The era of one-size-fits-all inference accelerators is ending the same way the era of one-size-fits-all CPUs ended in 2008 — through phase-aware specialization, not raw speed. The 2016 Xeon isn't a curiosity. It's the first node in a heterogeneous inference farm that doesn't have a name yet.