Do transformers really need three projections? New paper says no

What happened

A paper posted to arXiv (2606.04032) under the title *Do transformers need three projections? Systematic study of QKV variants* ran the experiment everyone has been meaning to run for five years. The authors trained transformer models from 125M to 1.3B parameters across a controlled grid of attention variants: the standard three-projection QKV, two-projection variants where Q and K share weights (or K and V share weights), and single-projection variants where one matrix is reused for all three roles with learned per-role biases or low-rank deltas.

The headline result: at every scale they tested, the two-projection variant where K and V are tied matched full QKV within noise on validation perplexity, while cutting attention parameters by roughly a third. Single-projection variants degraded measurably but recovered most of the gap with a small low-rank delta per role — a configuration the authors call "QKV-LoRA-at-init." The ablations cover GPT-style decoder-only training on a SlimPajama subset, with matched token budgets and matched optimizer schedules.

The paper sits in a lineage that includes Multi-Query Attention (Shazeer, 2019), Grouped-Query Attention (Ainslie et al., 2023), and Multi-Head Latent Attention (DeepSeek-V2). All of those compress K and V *across heads*. This paper asks the orthogonal question: do you need separate matrices to *produce* K and V in the first place, before the head-sharing happens?

Why it matters

For the last three years, every serious LLM team has been grinding on the same bottleneck: KV cache memory. At inference, attention math is cheap — it's the cache of past K and V tensors that blows up GPU memory and forces batch sizes down. GQA cut that by sharing K and V across query head groups. MLA cut it further by projecting to a latent space. The default assumption underneath all of this work was that K and V are fundamentally different objects that need their own projections — K for similarity, V for the payload.

This paper challenges that assumption empirically. If K and V can share a projection without quality loss, the implications cascade in two directions. First, training: a third of the attention block's parameters disappear, which at Llama-3-70B scale is roughly 6B parameters saved with no architecture change beyond a weight tie. Second, inference: combined with GQA or MLA, you compound the savings. A KV-tied + GQA-8 model has roughly half the attention parameters and half the cache memory of a vanilla MHA model at the same hidden dim.

The HN thread (134 points, 80+ comments at time of writing) is unusually substantive. The top-voted skeptic flagged that the scaling curves only run to 1.3B, and that prior "this works at small scale" attention tweaks (Linformer, Performer, Synthesizer) have a long history of breaking past 7B. That critique is fair: the paper's strongest claim is "no degradation at 1.3B," not "safe for production at frontier scale." A second commenter pointed out that the paper doesn't ablate against MLA, which is now the state-of-the-art KV compression on open weights — so the comparison baseline is arguably one generation behind.

But the methodology is what makes this paper land. Most architecture papers in this space run one configuration, hit a benchmark, and declare victory. This one runs the full Cartesian product — 4 sizes × 6 attention variants × 3 seeds — and reports the raw curves, not just the best number. That's the kind of paper a research lead can actually act on without having to re-run everything from scratch.

What this means for your stack

If you train your own models — and a growing number of teams now do, given that base-model fine-tuning has gotten cheap — the practical action is small: try the K-V tied variant on your next pretraining run. The change is roughly ten lines in a transformer block, the parameter count drops by a measurable fraction, and if the paper holds, your loss curves should overlap with the baseline. At minimum, this is a cheap experiment with a known downside (slightly worse loss) and a known upside (smaller model, smaller cache).

If you serve models, the immediate impact is zero — Llama 4, Qwen 3, and DeepSeek-V3 all already ship. But the next-generation open-weight releases are being trained right now, and architecture choices made in the next six months will be locked in for years of serving. Push your model providers (or your internal training team) to A/B this variant in their next run. The cost-per-million-tokens math at serving scale rewards any KV reduction disproportionately, because cache memory is what determines how much batching you can do, and batching is what determines throughput.

For the framework layer, the implications are mostly mechanical. vLLM, SGLang, and TensorRT-LLM all assume separate K and V projections in their attention kernels. A KV-tied model would need a one-line config plus a minor kernel tweak — nothing structural. The risk is more that downstream tools (LoRA adapters, fine-tuning libraries, quantization toolchains) all have implicit assumptions about the QKV split that would need a sweep to update.

Looking ahead

The real test is whether someone trains a frontier-scale (70B+) model with KV-tied projections and publishes the eval suite. That experiment costs seven figures, which is why nobody has done it, which is why papers like this one matter — they de-risk the bet enough that a well-funded lab might actually run it. If DeepSeek or Qwen ship a KV-tied variant in their next release and the MMLU numbers hold, the default transformer block changes industry-wide within a year. If the result fails to scale, this becomes another Linformer — a clean idea that didn't survive contact with frontier compute. Either way, the next interesting attention paper isn't going to be about Q, K, or V — it's going to be about whether you need them as separate concepts at all.