The Kimi team frames KDA as the first linear-attention variant to outperform full attention in a matched-training comparison (same tokenizer, same 1.4T-token budget, same architecture family). They back this with concrete gains: up to 75% KV cache reduction and 6× faster decoding at 1M-token context against an MLA baseline, while also improving long-context quality rather than trading it away.
By submitting the paper with its self-descriptive title 'An Expressive, Efficient Attention Architecture,' the submitter amplifies the claim that this hybrid design is both expressive enough to preserve quality and efficient enough to matter at scale. The 228-point score suggests strong community endorsement of the framing that this is a substantive advance.
The editorial argues that KDA's key contribution is replacing the scalar forget gate of prior gated delta rule work (DeltaNet, Gated DeltaNet, Mamba2) with a diagonal gate per feature channel, giving the recurrent state finer control over memory retention. This — combined with a chunkwise parallel algorithm that makes it trainable at scale — is what elevates it above the usual 'linear attention curiosity' pattern.
The editorial explicitly frames linear attention as 'the perpetual bridesmaid of the transformer world for five years,' noting that every six months a new variant claims parity with softmax attention and then quietly loses on long-context recall or in-context learning. The implicit position is that even Kimi Linear's impressive numbers should be treated cautiously until independent replication at frontier scale confirms the matched-training result holds up.
Moonshot AI (the Kimi team) published Kimi Linear, a hybrid attention architecture that mixes a new linear-attention variant called Kimi Delta Attention (KDA) with standard full attention at a 3:1 ratio — three KDA layers for every one full-attention layer. The paper reports it as the first linear-attention hybrid to outperform full attention under a fair, matched-training comparison: same tokenizer, same 1.4T-token training budget, same architecture family otherwise.
The headline numbers are the ones practitioners will care about. Against a pure MLA (Multi-head Latent Attention) baseline of the same size, Kimi Linear reduces KV cache memory by up to 75% and delivers up to 6× faster decoding throughput at 1M-token context. Short-context quality holds; long-context quality actually improves on several benchmarks, which is the reverse of the usual linear-attention trade.
KDA itself is a refinement of the gated delta rule line of work (DeltaNet, Gated DeltaNet, Mamba2). Instead of a scalar forget gate, KDA uses a fine-grained diagonal gate per feature channel, giving the recurrent state finer control over what to remember and what to overwrite. The team also ships a chunkwise parallel algorithm — the reason it trains at scale instead of being a curiosity — and the kernel has already landed upstream in the Flash Linear Attention (FLA) library. Base and instruct checkpoints (48B total / 3B active MoE) are on Hugging Face.
Linear attention has been the perpetual bridesmaid of the transformer world for five years. Every six months a new variant claims to match softmax attention, and every time it quietly loses on long-context recall, in-context learning, or both. The playbook has been: publish, benchmark on small models, ignore the perplexity gap, move on. Kimi Linear is the first result in a while where the matched-training comparison actually favors the linear side — and it's at a scale (1.4T tokens, MoE) that maps to how frontier models are actually built.
The 3:1 hybrid ratio is doing a lot of the work here, and it's worth being honest about that. Pure linear attention still can't do everything softmax attention can — associative recall degrades, needle-in-a-haystack gets shakier, and certain in-context reasoning patterns need the full pairwise interaction. Interleaving one full-attention layer every four layers gives the model a periodic 'global sync' while keeping the memory and compute footprint dominated by the cheap KDA layers. It's the same intuition behind Jamba, Zamba, and Samba — hybrid stacks that use Mamba/linear layers for bulk and attention for precision — but with a cleaner ablation and a stronger recurrent primitive.
The per-channel gating in KDA is the technical detail that likely matters most for downstream builders. Gated DeltaNet already showed that a single scalar gate improves over vanilla delta rule; Kimi's move to a diagonal gate is a small architectural change with an outsized effect on how much state the model can maintain per token. Community reaction on Hacker News has been unusually specific — commenters are already asking whether the KDA block can be swapped into existing Llama-family checkpoints via distillation, which is the right question. If a 3B-active MoE with 75% less KV cache genuinely matches a full-attention model of the same class, the economics of serving long-context inference change more than any single decoding trick this year.
The cache reduction is the number to focus on. KV cache is what makes 1M-context serving expensive — not FLOPs, memory bandwidth and HBM footprint. A 75% cut means either 4× more concurrent long-context sessions per GPU, or the same sessions on cheaper hardware, or both. The 6× decode speedup at 1M compounds on top of that. For anyone running RAG-heavy workloads, agentic loops with big scratchpads, or code-assist over whole repositories, this is the direction the cost curve has been waiting to bend.
If you're training your own base models, Kimi Linear is now the reference architecture to beat for anything long-context. The kernel is in FLA, the ratio is published, and the ablations cover enough of the design space that you're not guessing. Expect the next round of open-weight releases from the usual suspects — Qwen, DeepSeek, Mistral, possibly Meta — to at least evaluate a hybrid stack, and probably ship one.
If you're serving inference, the practical move right now is smaller: pull the checkpoints, benchmark them against whatever you're currently running for long-context workloads, and measure the real KV footprint on your traffic shape. The paper's numbers are on synthetic long-context evals; your mileage on real agent traces or long RAG documents will vary. But if the cache reduction holds anywhere close to advertised, the vLLM/SGLang integration path is short — FLA is already a known dependency in the serving stack.
The one caveat worth flagging: hybrid architectures have historically been harder to quantize and harder to fine-tune cleanly, because you have two different attention mechanisms with different numerical properties. INT4 support, LoRA fine-tuning on the KDA layers, and Speculative decoding compatibility are all things that need to be verified before this replaces a well-worn Llama-style backbone in production. Give it a quarter for the ecosystem to catch up.
The interesting question isn't whether Kimi Linear itself becomes the new default — it's whether the hybrid + fine-grained-gating pattern does. If the next Kimi frontier model ships with this architecture and posts competitive numbers against GPT and Claude on long-context reasoning, the debate about linear attention is over. Every serious base-model team will have a hybrid stack on the roadmap by Q1 2027, and the long-context pricing gap between open and closed models narrows fast. If it turns out the 3:1 ratio doesn't scale to 100B+ dense parameters or breaks under RLHF, we've still learned something useful about how much full attention you actually need. Either way, this is the first linear-attention paper in a while that's worth reading end to end.
Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).[0] https://arxiv.org/abs/2607.24653
if non standard transformers like this take off are companies like etched forked?
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.This is just awesome.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?Like isn't it weird that the 1 million parameter model with the same architecture can&#x