Piffer's hand-authored benchmark uses competition-difficulty problems with fresh structures that don't exist online, and frontier models drop dramatically compared to their public scores. He argues this proves benchmark wins measure how much of the internet's math corpus a model has absorbed, and introduces a taxonomy including 'structural leakage' where models pattern-match to isomorphic problems without reasoning.
The editorial argues the IMO-gold headlines weren't the phase transition they appeared to be — they're closer to a very good open-book exam than mathematics. This matters because vendors have quietly folded 'can do olympiad math' into broader 'can reason' claims used to justify agent architectures, autonomous coding, and formal-verification pitches, drawing an analogy to Deep Blue's narrow chess mastery.
Piffer's data shows that perturbing a known problem (changing numbers or wording) barely affects model performance, but genuinely novel problem structures wreck it. He uses this asymmetry to argue that current benchmark methodology — which relies heavily on perturbation to defend against contamination — systematically underestimates memorization effects.
Davide Piffer, a researcher who has spent the last two years poking holes in LLM math evaluations, published a fresh benchmark this week that does something surprisingly rare: it uses problems that aren't in the training data. He hand-authored a set of competition-style questions — same difficulty band as AIME and IMO shortlist problems — but with fresh numbers, fresh geometry, and fresh combinatorial setups that don't appear anywhere Google can find them.
The results are the story. Models that post IMO-gold-adjacent scores on the standard math benchmarks drop dramatically on Piffer's private set. He reports frontier models solving a fraction of what their public scores would predict, with the biggest gaps on problems whose *structure* — not just numbers — is unfamiliar. Perturbing a known problem barely dents performance. Writing a genuinely new one wrecks it.
Piffer's conclusion is blunt: the state-of-the-art on math benchmarks is largely a measure of how much of the internet's math corpus a model has memorized, not how well it reasons. The paper walks through a taxonomy of contamination — direct leakage, solution-set leakage, and what he calls "structural leakage," where the model has seen a problem isomorphic to the test one and pattern-matches its way to the answer without ever doing the reasoning steps.
The IMO-gold headlines from July felt like a phase transition. They weren't. What Piffer's work makes clear is that the celebrated math wins are closer to a very good open-book exam than they are to a mathematician doing mathematics. That distinction matters because the industry has been quietly folding "can do olympiad math" into a broader claim of "can reason," and using that claim to justify agent architectures, autonomous coding pipelines, and formal-verification pitches.
The uncomfortable comparison is chess engines circa 1997. Deep Blue beat Kasparov, but nobody claimed Deep Blue could generalize its search to unseen game families. The current LLM discourse blurs that line constantly. A model that gets 90% on MATH and 40% on a comparable never-seen set isn't doing 90% of the reasoning work — it's doing 40% of the reasoning work and 50% of the retrieval work, and the benchmark can't tell you which is which. The mathematicians in the comments on the HN thread — the ones who've actually graded olympiad papers — largely back Piffer's framing. Terence Tao's earlier remarks about LLM proofs being "mediocre-graduate-student" quality look, in retrospect, more generous than deserved once you strip out the memorization signal.
The methodology also validates a criticism that's been circulating for a year: benchmark saturation is not the same as capability. GSM8K got saturated. MATH is getting saturated. Every new frontier release cites gains on these, and every release quietly benefits from the fact that the test set has been in training corpora for years. Piffer's contribution is a reproducible recipe for producing contamination-free tests, not just this one round of results. Any team that wants to run it — internal red-teams, academic groups, standards bodies — now has a template.
The interesting second-order effect is what this does to the reasoning-model narrative. The o1-style and R1-style architectures were sold as reasoning breakthroughs partly on the strength of their MATH and AIME jumps. If a chunk of those jumps is actually improved retrieval of solution patterns under longer chain-of-thought (rather than novel search), the story about "test-time compute unlocks reasoning" needs a haircut. It doesn't disappear — the models really are doing *something* better than pure next-token prediction on hard problems — but the size of the delta is probably 30-50% of what the marketing implies.
If you're building anything that touches novel-input reasoning — agents that plan over your specific codebase, models that debug production issues they've never seen, systems that generate proofs against your specific schema — you should stop taking public benchmark scores at face value. Build a small private eval that mirrors your actual domain and, critically, that your model has provably never seen; the delta between that and the public number is the honest measure of what you're deploying.
Concretely: write 20-50 tasks from scratch, don't post them anywhere, and re-run them every model release. Version them. Rotate them. If your private eval shows a 30-point gap versus the public number, that gap is your risk budget for production surprises. It's also the number to put in front of the exec who wants to replace a human workflow with an agent because "GPT-5 gets 95% on the benchmark."
The second implication is for anyone selling AI into technical verticals — legal, medical coding, tax, engineering compliance — where the value prop hinges on the model handling novel edge cases. Piffer's methodology transfers directly. If your vendor won't run their model against a held-out set you author, that's a signal. If they will, and the numbers hold up, that's a much stronger sales artifact than any MMLU screenshot.
The next 12 months will separate labs that train against contamination-controlled evals from labs that keep chasing headline numbers on saturated benchmarks. Expect at least one frontier lab to quietly publish a "held-out reasoning" score alongside their next model card — the pressure is now high enough that pretending Piffer's critique doesn't exist looks worse than showing a lower but honest number. The mathematicians were right early; the benchmarks are finally catching up to them.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.