Janas streams MoE experts off SSD so your 70B fits in 16GB of RAM

What happened

A GitHub project called Janas (`prabanta-dev/janas`) hit the Hacker News front page with 130+ points on a one-line pitch: run LLMs larger than your RAM by streaming experts from SSD. The repo targets mixture-of-experts (MoE) models — the architecture behind Mixtral, DeepSeek-V3, Qwen3-MoE, and most of the frontier open-weights releases in the last 18 months — and treats the expert tensors as a swap space rather than a working set.

The core observation is not new, but the packaging is. In a MoE model, each token only activates a small fraction of the parameters — typically 2 of 8 experts per layer, sometimes less — which means the vast majority of weights sit idle on any given forward pass. DeepSeek-V3 is 671B total parameters but routes just ~37B through the network for each token. Mixtral 8x7B is nominally 46.7B parameters but computes against ~12.9B per token. The gap between "parameters you own" and "parameters you touch" is the arbitrage Janas is trying to exploit.

The implementation reads expert weights from NVMe on demand, keeps a hot cache of recently-used experts in RAM, and lets the OS page the rest. On paper, it means a 70B-class MoE can run on a laptop with 16GB of RAM — provided the SSD can feed the model fast enough to keep the GPU (or CPU) from stalling.

Why it matters

For the last year the local-LLM community has been bending over backwards to shrink models into consumer memory: aggressive quantization (GGUF Q2, Q3), speculative decoding, KV cache offload, llama.cpp's mmap tricks. All of these treat RAM as the hard constraint and burn accuracy to fit.

Janas inverts the question: instead of making the model smaller, treat the storage hierarchy as the model's working memory and let the router decide what to load. This is closer in spirit to how operating systems handle virtual memory than how inference engines usually work. The relevant prior art is PowerInfer (which classifies neurons as hot/cold and keeps hot ones in VRAM), DeepSpeed-Inference's ZeRO-Inference offload, and Microsoft's EDGE/LLM-in-a-Flash paper from Apple research — the latter being the most direct intellectual ancestor, since Apple showed in late 2023 that you could run a 7B dense model from flash with reasonable throughput using sliding-window prefetching and sparsity prediction.

What's different now is that MoE architectures make the sparsity *explicit and architectural* rather than *latent and learned*. You don't have to predict which neurons will be active — the router literally tells you which experts to load, every token, every layer. That's a much friendlier access pattern for a disk-backed system than the stochastic sparsity of a dense ReLU model.

The real benchmark is tokens per second per gigabyte of SSD bandwidth, not perplexity or MMLU. A PCIe 4.0 NVMe tops out around 7 GB/s sequential; PCIe 5.0 drives push 12-14 GB/s. If each decoded token needs ~100MB of expert weights pulled from disk (a reasonable estimate for a quantized 70B MoE with 8 experts per layer × 32 layers × 2 experts active × a few hundred MB each, before caching), you're looking at a theoretical ceiling in the single-digit tokens/second range — before cache hits, which should be substantial because routing has strong temporal locality within a conversation.

Community reaction on the HN thread splits predictably. One camp notes that this is roughly what llama.cpp's `--n-gpu-layers 0 --mmap` already does for dense models, with the OS page cache doing the heavy lifting — and that MoE-aware prefetching is the real contribution, not the offload itself. The other camp points out that any serious user already has a GPU and would rather spend money on a used 3090 than wait for NVMe latency. Both are correct, which is the usual shape of a useful tool: it's not for everyone, but for a specific slice of users — people with a fast SSD, no 24GB GPU, and a willingness to accept 2-5 tokens/sec — it unlocks model classes that were previously unreachable.

What this means for your stack

If you run local inference on a Macbook or a workstation without a big GPU, this is worth tracking. The practical calculus: a Framework laptop or recent Mac with a Gen4 NVMe and 16-32GB of unified memory is now a plausible host for DeepSeek-V2-Lite, Mixtral 8x7B at FP8, or Qwen3-MoE variants — models that would otherwise demand an A6000 or dual 3090s. For agent workloads where you care more about model capability than latency — overnight batch summarization, local code review, offline RAG — a 2 tok/sec ceiling on a 70B-class MoE beats a 50 tok/sec floor on a 7B dense model nearly every time.

If you're building a product that ships with local inference (think Jan, Ollama, LM Studio, or anything embedding a model into a desktop app), SSD-backed MoE changes your sizing matrix. You can target users with 16GB of RAM and still ship meaningfully capable models — the trade is now disk space (which users have in abundance) and tolerance for slower generation (which depends on the use case). The failure mode to watch for: thermal throttling on consumer NVMe drives under sustained read load. Enterprise drives rated for mixed workloads will handle this; cheap QLC drives will cook themselves and degrade.

For infrastructure teams, the broader implication is that the storage-tier matters again for inference. For two years the mental model has been "GPU memory or bust." MoE offload reopens the DRAM and NVMe tiers as legitimate parts of the inference pipeline — which is going to make capacity planning more interesting, especially as frontier models trend larger and sparser.

Looking ahead

The interesting question is whether this stays a hobbyist tool or whether the big inference engines (vLLM, SGLang, TensorRT-LLM) absorb the idea. MoE-aware expert offload is the kind of optimization that makes economic sense for cloud providers too — if you can serve a 500B-parameter model from a node with 128GB of DRAM instead of 8xH100s, your margins look very different. The research here is already landing: expert prefetching, speculative routing, and expert pinning based on conversation context are all open problems with real papers behind them. Janas is early, probably rough, and will get copied. But the direction is right, and the direction is where frontier inference is going anyway.