Flash-MoE Fits a 397B Model Into 48GB of Mac RAM

A new open-source project called Flash-MoE demonstrates something that would have sounded absurd eighteen months ago: running a 397B parameter model on a MacBook with 48GB of unified memory.

The trick, of course, is that it's a Mixture-of-Experts model — and MoE architectures are uniquely suited to memory-constrained inference. A 397B MoE model doesn't activate all 397 billion parameters for every token. It routes each token through a small subset of expert networks, meaning only a fraction of the total weights need to be in fast memory at any given moment. Flash-MoE exploits this by keeping only the active experts in RAM and offloading the rest, effectively trading compute latency for memory savings.

This is the same architectural property that makes models like DeepSeek-V3 and Mixtral interesting: massive total capacity with sparse per-token compute. What Flash-MoE adds is a practical implementation that targets Apple Silicon's unified memory architecture specifically. The M-series chips blur the line between CPU and GPU memory, which eliminates the PCIe bottleneck that makes expert offloading painful on traditional setups.

The Hacker News crowd (231 points) is understandably interested. Local inference has gone from a hobbyist curiosity to a legitimate deployment option for many use cases — privacy-sensitive workloads, offline scenarios, development iteration loops where API latency kills flow. Being able to run a model of this scale on hardware that fits in a backpack changes the calculus on what 'local-first AI' can mean.

Some important caveats apply. Tokens-per-second on a 48GB machine running a model this size won't compete with cloud inference — you're almost certainly looking at single-digit tok/s at best, possibly sub-1 for long contexts. The memory headroom is razor-thin, meaning you can't run much else alongside inference. And MoE models, while parameter-efficient at inference time, still need careful quantization (likely 4-bit or lower) to fit within the memory budget.

But that's somewhat beside the point. The significance is directional: the gap between 'cloud-only' model sizes and 'runs on my laptop' model sizes is closing faster than most people expected. Two years ago, running a 7B model locally was impressive. Last year, 70B became feasible on high-end consumer hardware. Now we're looking at nearly 400B — albeit with the MoE caveat.

For practitioners, Flash-MoE is worth watching as a reference implementation for efficient MoE inference on Apple Silicon. If you've got an M2 Pro/Max/Ultra or M3/M4 with 48GB+ unified memory and want to experiment with large-scale local models, this is the tooling to try. Just don't expect ChatGPT-speed responses — the point is that it runs at all.