Franke argues that the headline context size on a model card only tells you what the API will accept, not what the model can reliably reason over. Drawing on benchmarks like NoLiMa and RULER, he shows that retrieval accuracy collapses well before the advertised limit — often below 50% by 32k tokens once any inference is required — so trusting the printed number leads developers into hours of wasted debugging.
The editorial extends Franke's argument with an economic lens: input tokens are billed at full rate whether the model attends to them or not, so dumping a 400k-token repo to answer a question that needed three files means paying for ~397k tokens of noise that statistically drags the answer toward the mean. The implication is that model cards systematically under-disclose the 'effective context' region where retrieval stays above a usable threshold.
A faction in the HN thread pushed back that benchmarks like NoLiMa and RULER are contrived retrieval tests and don't capture workloads where context is coherent prose, structured code, or otherwise well-aligned with the model's attention patterns. They concede degradation exists but argue the synthetic numbers overstate how badly real applications suffer.
Commenters who build agents and RAG pipelines noted that anyone shipping production systems learned long ago to chunk, retrieve, and curate context rather than dump everything in. For them the post is validating what tooling like vector search and selective context assembly already assumes — the marketing number was never the working number.
A third subset of the thread admitted they had been pasting entire repositories into Claude or GPT and noticing the answers got worse as the repo grew — without connecting it to context-length degradation. Their reaction reframes the post as a needed public-service announcement rather than insider knowledge.
Garrit Franke's post "Don't trust large context windows" climbed to 214 on Hacker News this week — the second time in 36 hours a piece making the same argument has hit the front page. The thesis is unglamorous and now well-supported: the headline context size on a model card describes what the API will accept, not what the model will reliably reason over. Garrit walks through the gap with the bluntness of someone who's lost a weekend to it.
The corroborating data has been piling up for a year. NoLiMa (the "needle-in-a-needlestack" benchmark) showed that even frontier models drop below 50% retrieval accuracy somewhere between 32k and 128k tokens once the needle requires one hop of inference rather than a verbatim string match. RULER, the long-context evaluation from NVIDIA, found that models advertising 128k–1M windows often degrade to roughly half their short-context accuracy by 32k. The number printed on the model card is the size of the bucket; the question nobody answers on the card is how much of the bucket is structurally usable.
The HN thread, predictably, split into three camps: people who already knew (mostly agent builders), people who thought the benchmarks were unfair (a fair point — synthetic needle tests don't capture every workload), and people quietly admitting they'd been pasting entire codebases into Claude and wondering why the answers got worse the bigger the repo.
The practical consequence isn't "long context is fake." It's that long context is priced like memory and behaves like storage. Input tokens are billed at the full rate whether the model attends to them or not. If you load 400k tokens of repo context to answer a question that only needed the three files in `src/auth/`, you paid for 397k tokens of noise that statistically dragged the answer toward the mean.
This is the part the model cards don't print. Effective context — the region where retrieval accuracy stays above some usable threshold, say 80% — is usually 1/4 to 1/10 of the advertised window, and it lives at the beginning and the end, not the middle. The "lost in the middle" finding from Liu et al. is now two years old and has been reproduced across every frontier model that's been measured. It hasn't gone away with scale; it's gotten subtler.
The community reaction on HN was telling. The top comment thread was a quiet consensus among people shipping agents: nobody serious is using raw 200k+ context as a memory substitute in production. They're doing what people doing this seriously have always done — retrieval, summarization, hierarchical chunking, scratchpads, and aggressive context pruning between turns. The 1M-token window is, for them, a debugging convenience and a marketing line item, not an architecture.
The disagreement is about how to communicate this to teams that aren't shipping agents yet. One faction wants vendors to publish honest "effective context" curves alongside the marketing number; another argues the abstraction is the customer's problem, the same way nobody expects DRAM vendors to publish realistic working-set sizes. Both are right, in the sense that both are describing a market failure neither side is incentivized to fix.
There's also the cost asymmetry, which gets less attention than it deserves. A naive "paste the whole repo" workflow on Claude Sonnet at $3/M input tokens runs you $1.20 per 400k-token query. A retrieval-augmented version of the same workflow with a 16k working context costs $0.048 and — per every benchmark we have — answers more accurately. The 25x cost delta isn't buying you capability. It's buying you the comfort of not having to build retrieval.
If you're building anything that touches an LLM more than once per session, treat context as a budget, not a buffer. Three concrete moves:
Measure your own effective context. Run a needle test on your actual workload — not the synthetic NoLiMa one, but your data, your queries, your model. Insert a known fact at positions 5%, 25%, 50%, 75%, 95% of your typical prompt and measure recall. The number you get back is the only context size that matters for your application; everything past it is a billing line item.
Move retrieval before generation, not after. If you're stuffing a long context and asking the model to find the relevant part, you're making the model do retrieval inside attention, which is the most expensive retrieval mechanism ever invented. A 50ms vector search in front of a 16k prompt beats a 200k prompt on accuracy, latency, and cost simultaneously. The only reason teams don't do this is that pasting is easier than indexing — and that calculus changes the first time you look at a $4,000 monthly OpenAI bill.
Position the important stuff at the edges. If you must use long context, put critical instructions and key facts in the first 10% and last 10% of the prompt. The middle is for the model to skim. This isn't a hack; it's an empirical regularity that has held across every model family for two years.
The interesting question isn't whether context windows will keep growing — they will, because that number is easy to market — but whether anyone will publish a benchmark that becomes the de facto "effective context" number the way MMLU became the de facto reasoning number. Until then, the model cards will keep printing 1M, the bills will keep arriving in full, and the teams shipping reliable agents will keep quietly doing retrieval. The gap between what's advertised and what's usable is, increasingly, the moat — and it belongs to whoever bothers to measure it.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.