Tao argues the field has anchored on IMO/Putnam-style problems because they are cheap to grade with short verifiable answers, and this convenience has quietly bent the entire research program around a proxy that no longer tracks the underlying capability. He frames this explicitly as 'misalignment' in the AI-safety sense — the objective function has diverged from what would actually make AI useful to working mathematicians.
Tao points out that competition math looks objective because answers are short verifiable strings, but the reasoning producing them is never scored. Models can therefore output a correct boxed answer while fabricating lemmas, misciting theorems, and producing proofs a referee would bounce on first read — a failure mode invisible to the benchmark but disqualifying in actual research mathematics.
The editorial emphasizes that Tao has publicly used LLMs as a research aide, co-authored Lean formalization work, and endorsed Copilot-style tools for scut work. That track record is what makes the post land — the person best positioned to be an AI-for-math booster is instead saying the field's compass is pointing in the wrong direction, which is much harder to dismiss than a broadside from a skeptic.
The editorial draws the analogy to compiler benchmarks in the 2000s and ImageNet in the 2010s: a metric that was useful early on became the target itself, and once optimization pressure was applied, the metric stopped correlating with the underlying capability. Competition math is characterized as an especially treacherous instance of this pattern because its short-string answer format gives it a false veneer of objectivity.
Terry Tao — Fields medalist, UCLA professor, and one of the more prolific and even-handed voices on AI in mathematics — published a post titled *A Severe Misalignment of AI in Mathematics* on his blog. The framing is deliberate: misalignment, in the AI-safety sense. Not "the models are bad," not "the hype is overblown," but that the objectives the field is optimizing against have quietly diverged from the objectives that would actually make AI useful to working mathematicians.
Tao's argument is that the community has anchored on competition-style benchmarks — IMO problems, Putnam-style contests, closed-form answer keys — because they are cheap to grade, and this convenience has bent the entire research program around them. The result is a generation of models that can produce a plausible-looking write-up of an olympiad problem while still fabricating lemmas, misciting theorems, and producing proofs that a referee would bounce on first read.
The post is not a broadside. Tao has been publicly positive about using LLMs as a research aide, has co-authored formalization work with Lean, and has talked openly about using Copilot-style tools for scut work. That is what makes the piece land: the person best-positioned to sing the praises of AI-for-math is instead saying the field's compass is pointing in the wrong direction.
The closest analogy for a working engineer is what happened to compiler benchmarks in the 2000s, or to ImageNet in the 2010s: a benchmark that was useful early on became the target itself, and optimizing for the target stopped correlating with the underlying capability. Competition math is an especially treacherous benchmark because the answers are short, verifiable strings, which makes it look objective, while the reasoning that produced them is not scored at all.
This matters because the failure mode of a wrong proof is not the failure mode of a wrong answer. A model that answers `\boxed{42}` correctly on an IMO problem while producing a proof that hand-waves through the crux is scored the same as a model that produces a rigorous argument. In research, only the second one is worth anything. A referee is not looking at the final integer; they are looking at whether every step is derivable, whether the lemmas invoked actually say what you claim they say, and whether the counterexamples you dismissed are really dismissable.
Tao's second concern — and this is where the piece gets uncomfortable for the eval community — is contamination and Goodharting. Olympiad problems and their solutions have been on the open web for decades. When a lab reports that its new model "solves" some fraction of IMO 2024, the honest version of that number requires ruling out that the training corpus already contained walkthroughs of the same problems, or problems structurally similar enough that pattern-matching works. That is very hard to prove, and the incentive to prove it is low when the headline number is the fundraising asset.
The community reaction on Hacker News mirrored what you would expect: a mix of "yes, and this generalizes to every AI benchmark, not just math," and "but the models really are getting better at some verifiable tasks, isn't that progress?" Both are true. The reconciliation is that being better at a narrow verifiable task is a real capability gain, but it is not evidence of the general capability the marketing implies. Solving more contest problems is not the same as being able to help prove something that has never been proved before, and the field has been sloppy about equating the two.
There is also a quieter point buried in Tao's post that deserves lifting out. Real mathematical research is closer to software engineering than to competition programming. The interesting work is not in cracking a well-posed puzzle in four hours; it is in the months of formulating the right question, discarding formalisms that do not survive contact with edge cases, chasing down a citation trail, and — increasingly — writing Lean or Coq that a machine can check. The AI capability that would actually change research is being able to formalize an informal argument, spot a gap in a preprint, or search the literature for a lemma you half-remember. None of that is on any leaderboard.
If you are building or buying AI evals for anything — code review, security triage, SRE runbooks, legal drafting — Tao's critique reads as a general warning. The three failure modes generalize cleanly.
First, answer-only grading hides reasoning failures. If your eval only checks the final artifact — did the PR merge, did the alert get closed, did the contract clause get written — you are measuring pattern-match completions and calling it competence. Grade the intermediate reasoning, or accept that you will ship agents that guess right for the wrong reasons and then guess wrong catastrophically when the distribution shifts.
Second, contamination is not a math problem, it is an everything problem. Any benchmark drawn from the public web is, by 2026, functionally part of the training set. This does not mean benchmarks are worthless, but it does mean that a delta of a few percentage points between two frontier models on any published eval is inside the noise of "who scraped harder." If you are making procurement decisions on public benchmarks alone, you are being sold a number that both vendors optimized for and neither would let you audit.
Third, the useful capability is rarely the one being reported. For math it is formalization and gap-finding; for code it is the boring middle of a refactor, not the eye-catching "write me a Tetris clone" demo; for security it is triage volume at low false-positive rates, not one-shot CVE-solving stunts. Build your internal evals around the thing that would actually save you an engineer-week, and treat the vendor leaderboard as a marketing artifact.
The interesting question is whether the AI-for-math community listens. There is money and prestige in the current benchmark culture, and the alternative — evaluating models on real research contributions, formalization throughput, or referee-quality proof-checking — is expensive, subjective, and slow. Tao's post is a marker: when the field's most credible voice publicly says the target is wrong, the labs that keep pointing at olympiad scores as evidence of research-grade capability are going to look increasingly like they are performing for the wrong audience. The same reckoning is coming for every domain where AI evals have gotten easy to publish and hard to trust.
> solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight.This is the effect of AI on most intellectual disciplines, and it’s a real worry.
On the contrary to what Tao believe, it seems like we need AI to move the needle on mathematics.> problems in many fields of mathematicsDeveloping these different fields moves complexity from the field itself to the interactions of these fields.Getting too preoccupied with the established termino
If you internalise that AI might actually reach super intelligence then logically the question becomes "so what exactly are humans for if literally everything can be done better by machines?". Then mathematics and all intellectual work, as argued for here, becomes quite clearly a recreatio
I always wondered how Idiocracy got to the point where they have sophisticated technology and yet everyone is stupid. I think we have our answer.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
To me it doesn't seem like what AI has destroyed is the ability for mathematicians to develop understanding and share it with each other, but rather it's destroyed the yardstick (solving open problems) that has traditionally been used to measure how much they have contributed to that under