Apple's team argues the key breakthrough isn't speed but scale — by sharding models across multiple Mac Studios over Thunderbolt or Ethernet, users can run hundreds-of-billions-of-parameters models that previously required rented H100 clusters. They emphasize the model-parallel partitioning keeps the user-facing API unchanged from single-device MLX.
The editorial highlights Apple-specific advantages often overlooked by CUDA-native developers: unified memory eliminates host-to-device copies within a node, Thunderbolt 5's 80 Gbps symmetric links dwarf typical self-hoster Ethernet, and setup is a single pip install with no driver or CUDA version hell. These advantages compound to make multi-Mac clusters uniquely practical.
The editorial frames the economics as the real story: ~$20K buys two maxed-out Mac Studios that sit silently under a desk and can run 405B-class models, versus six-figure capex plus datacenter contracts for an H100 node or $4–8/hr indefinitely on spot. This collapses the asymmetry where open weights were downloadable but not actually runnable outside rented GPU clusters.
By submitting the Apple ML Research post to Hacker News where it reached 225 points and 104 comments, antipaul signaled that the self-hoster community sees this as a meaningful shift. The high score indicates broad interest in a credible path to running the largest open models without datacenter infrastructure.
Apple's machine learning research team shipped distributed inference support in MLX, the Apple-silicon-native array framework they open-sourced in late 2023. The new capability lets a single model be partitioned across multiple Macs — connected over Thunderbolt 4/5 or plain Ethernet — and run as if it were on one machine. The write-up focuses on Mac Studio as the reference node: an M2 Ultra or M3 Ultra with up to 512GB of unified memory on the top-end M3 Ultra configuration.
The headline isn't that MLX got faster — it's that "the biggest open-weights model you can run" is no longer bounded by what fits in one box. Apple demonstrates sharding across two Mac Studios to serve models in the hundreds-of-billions-of-parameters range at usable token rates. The partitioning strategy is model-parallel: layers (or layer groups) are placed on different nodes, activations flow between them over the interconnect, and the user-facing API is essentially unchanged from single-device MLX.
Under the hood, this leans on a few Apple-specific advantages that are easy to forget if you've only worked on CUDA. Unified memory means there is no host↔device copy inside a node — the ANE, GPU, and CPU all see the same bytes. Thunderbolt 5 gives you 80 Gbps symmetric per link (120 Gbps asymmetric), which is not NVLink but is dramatically better than the 10/25 Gbps Ethernet most self-hosters have. And the whole thing boots off macOS with `pip install mlx` — no drivers, no CUDA toolkit version mismatches, no `nvidia-smi` roulette.
For two years the open-weights story has had an awkward asymmetric: you could *download* a 405B model but you couldn't *run* it anywhere that wasn't a rented H100 cluster. A pair of maxed-out Mac Studios runs you roughly $20K and sits silently under a desk — the equivalent H100 node is a six-figure capex plus a datacenter contract, or $4–8/hr indefinitely on the spot market. That's not an argument that Apple silicon beats H100s on throughput. It doesn't. It's an argument that for a very large class of workloads — internal tools, batch jobs, research prototypes, privacy-sensitive inference — throughput was never the constraint. Memory capacity and operational simplicity were.
The community reaction on Hacker News (225 points, mostly from people who have actually tried running big models at home) tracks this closely. The top comments aren't about tokens/sec. They're about not having to touch CUDA, about being able to keep weights on the same box that holds the data, and about the fact that a Mac Studio draws roughly 300W under load versus 700W per H100 before you count the rest of the chassis.
It's also worth being honest about what this is *not*. Distributed MLX is model-parallel, not tensor-parallel in the Megatron sense, and it won't give you the per-token latency of a well-tuned vLLM deployment on NVIDIA hardware. Prompt processing — the compute-heavy prefill phase — is still where Apple silicon trails. If you're serving a chatbot with p99 latency SLOs, this is not your production stack. If you're generating overnight embeddings, doing synthetic data generation, or running an internal coding assistant where 30 tok/sec is fine, the math changes entirely.
The other quiet win here is licensing clarity. MLX is MIT. Models are whatever the model vendor says they are. There is no CUDA EULA, no datacenter-use restriction, no "must use with approved partners" clause lurking anywhere. For enterprise teams that have been doing legal review on their inference stack, that's a surprisingly big deal.
If you're already running Ollama or llama.cpp on a single Mac, nothing changes today — but your ceiling just moved. The models you've been quantizing aggressively to fit in 128GB can now live at higher precision across two nodes. Qwen 2.5 72B at FP16, DeepSeek-V3 at 4-bit, Llama 3.1 405B at 4-bit — all become plausible on hardware you can buy retail.
For platform teams, the interesting experiment is colocating inference with data. The main reason to send prompts to OpenAI or Anthropic is capability, but the main reason *not to* is almost always data residency or egress cost. A pair of Mac Studios in a locked closet is a weird answer to "where should our RAG pipeline run," and six months ago it would have been a joke. It's not a joke anymore — it's a $20K line item that eliminates a per-token bill and a data-exit review.
There are real sharp edges. Thunderbolt topology matters: you want a direct cable between the two Studios, not a hub. macOS updates occasionally break low-level networking in annoying ways. Model loading is slow — expect multi-minute cold starts for 400B-class models. And you still need someone who knows how to shard a model sensibly; the framework exposes the primitives but doesn't magically figure out the optimal split for your workload.
The strategic read is that Apple is quietly building the first credible non-NVIDIA story for self-hosted inference, and they're doing it by leaning on the one architectural advantage they have that NVIDIA structurally can't copy: unified memory at consumer price points. The next 12 months are going to tell us whether "a cluster of Mac Studios" is a cute hobbyist pattern or an actual production tier — and the answer probably hinges on whether Apple ships proper rack-mountable hardware or lets third parties do it. Either way, the frame has shifted. For the first time since the GPU era started, "I need to run a 400B model" has an answer that doesn't start with a Hopper.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.