Wafer's benchmark shows GLM-4.5 running at 2,626 tok/s/node on an 8-GPU MI355X node at roughly half the total cost of Blackwell. They argue the MI355X's 288GB HBM3e per GPU (2.3TB per node) lets the entire 355B-parameter MoE fit in a single node with tensor-parallel=8, eliminating the cross-node sharding overhead that Blackwell HGX would require.
By submitting the Wafer benchmark to HN with the headline framing '2x lower cost than Blackwell,' latchkey endorses the cost-per-token argument and helped drive it to 302 points. The submission surfaces the economic case that MI355X's memory-per-dollar advantage translates directly to serving cost.
The editorial argues that the standard 2023-era objections (missing kernels, CUDA-only FlashAttention, no fused ops) have quietly been resolved. vLLM, SGLang, and PyTorch now have working ROCm paths, and Wafer hit the headline number using stock vLLM + ROCm 6.4 with Hugging Face weights — no exotic Nvidia-only tooling like TensorRT-LLM or FP4 was needed to be competitive.
The editorial highlights that MI355X's 2.3TB of HBM per node lets a 355B MoE like GLM-4.5 hold weights in FP8 plus KV cache without any pipeline parallelism or cross-node NVLink domains. Because Blackwell would need 4-way tensor parallelism just to hold the weights, the memory-first architecture flips the economics for the specific workload class of model-size-dominated MoE inference.
Wafer's configuration — tensor-parallel=8 within a single node with zero pipeline parallelism — is a direct demonstration of the fit-in-one-node thesis. Their throughput number is achievable precisely because they avoided the cross-node communication tax that Blackwell deployments of the same model would incur.
Wafer.ai published a benchmark this week putting Zhipu AI's GLM-4.5 — a 355B-parameter MoE with 32B active — on an 8-GPU AMD Instinct MI355X node and measuring 2,626 tokens/second/node in sustained inference. Their headline claim: the same workload costs roughly 2x less than running it on Nvidia's Blackwell (B200) generation, once you account for GPU price, power, and the fact that the model fits in a single node without cross-node tensor parallelism.
The MI355X, announced by AMD at Advancing AI 2025, ships with 288GB of HBM3e per GPU and 8TB/s of memory bandwidth. A single 8-GPU node therefore carries 2.3TB of HBM — enough to hold GLM-4.5's weights in FP8 with headroom for KV cache at long context. Wafer's stack is ROCm 6.4 with vLLM, and they report the model running with tensor-parallel=8 inside the node and no pipeline parallelism at all.
The interesting number isn't the throughput — it's that they got there without touching the exotic parts of the Nvidia stack: no NVLink domain across nodes, no TensorRT-LLM kernel voodoo, no FP4. Just vLLM's ROCm backend and the model card weights from Hugging Face.
For two years the standard answer to "why not AMD?" has been some combination of: the software isn't there, the kernels aren't fused, FlashAttention is a Cuda dialect, and nobody has time to port. That answer was correct in 2023 and increasingly wrong in 2024. In 2026 it's mostly cope. vLLM, SGLang, and PyTorch all have working ROCm paths, and the MI300X generation has been quietly eating serving workloads at hyperscalers who don't tweet about it.
The MI355X changes the pitch from "comparable" to "structurally cheaper for a specific workload class." That class is large MoE inference, where model size dominates and the winning strategy is fit-in-one-node, not shard-across-eight. GLM-4.5 at 355B parameters would need 4-way tensor parallel across a Blackwell HGX just to hold the weights in FP8 — you're paying the all-reduce tax on every layer. On MI355X, TP=8 stays inside a single NVSwitch-equivalent Infinity Fabric domain and the cross-GPU chatter never touches a network card.
The cost math Wafer walks through is worth reading in full, but the shape of it: B200 street price is roughly $40K-45K per GPU, MI355X is landing at $25K-30K depending on volume, and the MI355X node draws less power per token served because the batch sizes it can hit without spilling KV cache are larger. When you multiply throughput-per-dollar by throughput-per-watt, you get a 2x delta that isn't marketing — it's arithmetic.
The community response on HN was unusually calm for an AMD-vs-Nvidia post: 302 points, few "but CUDA" replies, and multiple commenters volunteering their own MI300X production numbers. One thread from a mid-size inference provider claimed 60% of their serving fleet is now AMD, and the only Nvidia they're still buying is for training. That's a signal, not a benchmark, but it's the kind of signal that shows up in earnings calls two quarters later.
The caveat that matters: this is inference, not training. Nobody is claiming MI355X beats Blackwell on pretraining a 400B dense model from scratch. The training tooling gap (Megatron-LM, NeMo, the NCCL fabric) is still real, and the collective-comms performance on ROCm still trails NCCL on tightly-coupled workloads. But the world's compute bill is shifting from training to inference — Anthropic and OpenAI have both hinted their inference spend now exceeds training — and inference is where the memory-per-dollar equation flips.
If you're serving open-weight models — Llama 4, Qwen 3, DeepSeek, GLM — and your bill is dominated by GPU-hours rather than engineer-hours, you now have a real decision to make. The port to ROCm/vLLM is a weekend of work for most stacks; the savings are structural and recurring. That's the trade a CFO understands.
Specifically: if you're running any MoE larger than about 200B parameters on shared H100/H200 nodes and paying the tensor-parallel latency tax, MI355X's 288GB per GPU collapses your topology. TP=8 inside one node beats TP=16 across two nodes on almost every metric that matters — latency, throughput, tail P99, and cost. The Blackwell answer to this is NVL72, which is real and impressive but sold at a price that assumes you have no alternative.
For smaller shops, the practical move is: rent an MI300X or MI355X hour on Vultr, TensorWave, or Hot Aisle, port your serving container to the ROCm base image, and measure. If your tok/s/dollar improves by 40%+ (Wafer's number is 2x, most real workloads land somewhere between 1.4x and 1.8x), the migration pays for itself in weeks. If it doesn't, you've spent an afternoon and learned something.
The one place to be careful: quantization support is still narrower on ROCm. FP8 works, INT8 works, but the bleeding-edge stuff (FP4, AWQ variants, some GPTQ configs) either lags Nvidia by a release or isn't there. If your serving strategy depends on aggressive quant, benchmark before you commit.
The Blackwell/MI355X delta will narrow — Nvidia will cut prices, AMD will hit supply ceilings, and both will ship successors within twelve months. But the story here isn't about the specific chips. It's that the software moat Nvidia spent a decade building is finally porous enough that a mid-size startup can publish a credible 2x-cheaper benchmark on a Wednesday and nobody in the replies argues with the methodology. That's a different market than the one we had last summer.
While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.
I think we should make it illegal to not specify the quantization in the headline for these types of posts.
There’s noticeable accuracy degradation when they switched from fp8 to mxfp4
Not a new phenomena - performance per dollar has been fairly steadily exponentialling since 1900 or so1900 - 2010 https://www.thekurzweillibrary.com/exponential-growth-of-com...1939 - 2023 https://medium.com/@timventura/kurzweils-law-for-the-ai-age-...
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
Can you folks add performance per watt as a metric to these comparisons, I honestly want to understand where AMD fits in the stack in terms of actual performance to dollars. I have had talks with companies wanting to build data centers outside of US and find it hard to source anything Nvidia in suff