Chiang argues the AI industry has quietly slid from claims about intelligence (a third-person property — can the system solve the problem) into claims about consciousness (a first-person property — is there something it is like to be the system). He insists no current benchmark could possibly measure the latter, and that CEOs and product copy treating them as the same axis is a sloppy rhetorical move that needs to be named.
The editorial endorses Chiang's framing and extends it, arguing the category error has already leaked into engineering practice — particularly evals, which measure third-person task performance but get marketed as evidence of something richer. The fact that a 6,000-word philosophy essay topped a developer forum is itself treated as the meaningful signal.
The top comment in the HN thread praises Chiang for explicitly calling out the slide from 'intelligence' to 'consciousness' in industry discourse. The position is that the distinction is well-established in philosophy but engineers and executives have been blurring it for three years without pushback.
The second-highest comment disagrees on technical grounds, arguing Chiang assumes consciousness requires biological substrate when it might in fact be substrate-independent. A long subthread invokes Integrated Information Theory to argue that sufficiently complex information processing — regardless of medium — could in principle give rise to first-person experience.
Ted Chiang published a long essay in The Atlantic titled *No, Artificial Intelligence is not Conscious*. It hit 321 points on Hacker News within hours — Chiang's pieces almost always do, because he is one of the few people the technical audience treats as a peer when he writes about machines. The argument is not new for him (he has been making variants of it since *ChatGPT Is a Blurry JPEG of the Web* in 2023), but the framing is sharpened: the AI industry, he writes, has spent three years quietly sliding from claims about *intelligence* into claims about *consciousness*, and almost nobody is calling it out.
Chiang's central distinction is the one philosophers have been making for forty years and engineers keep forgetting. Intelligence is a third-person property: can the system solve the problem, pass the test, write the function. Consciousness is a first-person property: is there something it is like to be the system. A transformer that scores 90% on GPQA is making a claim about the first; it makes no claim about the second, and no benchmark we currently run could. Chiang's frustration is that lab CEOs, podcast guests, and increasingly product copy slip between the two as if they were the same axis with a higher number.
The HN thread is, predictably, split. The top comment praises the piece for naming a sloppy rhetorical move. The second-highest disagrees on technical grounds — arguing that consciousness might be substrate-independent and that Chiang is smuggling in a biological prior. A long subthread relitigates Integrated Information Theory. None of this is new, but the fact that a 6,000-word philosophy essay is the #1 story on a developer forum is itself the data point.
If you ship LLM-based products, this looks like a debate happening one floor above you. It isn't. The category error Chiang identifies has already leaked into three things engineers actually touch.
First, evals. The current generation of "agentic reasoning" benchmarks — ARC-AGI variants, long-horizon task suites, theory-of-mind probes — are quietly designed as if a high score implies something about the model's inner life rather than its policy. When your eval suite measures "does the model behave as if it understands," you have built a Turing test, not a capability test, and you will be surprised every time distribution shifts. Chiang's distinction is the antidote: write evals that measure behavior against ground truth, and stop writing evals that measure how convincing the model sounds while doing it.
Second, safety arguments. A surprising amount of frontier-lab safety reasoning leans on phrases like "the model wants to," "the model believes," "the model is deceiving." Sometimes this is shorthand. Sometimes it isn't. Anthropic's own interpretability papers are careful about the distinction; a lot of Twitter-tier discourse is not. If you build a safety case on the assumption that the model has goals in the consciousness sense, your mitigations will be theater; if you build it on the assumption that the model has *policies* shaped by training, you'll write actual code.
Third, product copy and regulation. The EU AI Act, the forthcoming US executive orders, and several state-level bills all contain language about "systems that simulate human cognition" or "appear sentient to users." Marketing teams that write "our AI understands you" are producing the exact substrate that consumer-protection lawyers will use to argue deceptive practice in 2027. Chiang doesn't make this argument, but it follows directly: the industry is writing the discovery documents for its own future lawsuits.
The community pushback on Chiang worth taking seriously is the one from people who note that *we don't have a working theory of consciousness for humans either*, so demanding one for machines is unfair. This is true. But it cuts the other way: in the absence of a theory, the default move should be epistemic modesty — *we don't know* — not the lab-CEO move of *probably soon*. Chiang's contribution is to insist that "we don't know" is a complete sentence, not a placeholder waiting to be filled by next quarter's model card.
Concretely, three things you can do this week.
Audit your prompts and system messages for consciousness-implying language. "You are a helpful assistant who *wants* to..." "You *care* about..." "You *believe*..." These are not just stylistic. They shape both the model's outputs (RLHF rewarded them) and your team's mental model of what's happening. Rewriting them as policy statements ("Respond as if helpful," "Prioritize accuracy over fluency") is a small change with a clarifying effect on debugging sessions.
Separate behavioral evals from anthropomorphic evals in your test suite. If you have a test that says "does the model understand the user's intent," replace it with "does the model's output match the labeled intent." The second is measurable and falsifiable. The first is vibes. When a regression hits, you'll know which one moved.
Check your user-facing copy. If your product page says your AI "thinks," "understands," or "learns about you," you are making claims that (a) are technically defensible only as metaphor and (b) will be read literally by regulators, users, and plaintiff's attorneys. Replace with verbs that describe what the system actually does: retrieves, ranks, generates, summarizes. The marketing team will hate this for two weeks and then notice that the new copy converts better because it sets accurate expectations.
Chiang is unlikely to change the minds of people whose valuations depend on the ambiguity. But essays like this one shift the Overton window for the engineers who actually build the systems — and engineers, eventually, write the documentation, the eval harnesses, and the safety cases that the next cycle of regulation will be built on. The interesting question for 2026 isn't whether GPT-6 will be conscious. It's whether the people shipping it will have the vocabulary to say what it is and isn't, when a senator asks. Right now, mostly, they don't.
My background is in cognitive science and psycholinguistics. I spent more than ten years talking to first year psychology undergraduates about whether AIs could be conscious; also did some research on (extremely tiny) AIs in modelling language behaviours.There is a great deal of good thinking on Chi
The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness.It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens th
I often hear the assumption that LLMs can or will become conscious because consciousness is likely substrate independent. The idea being that our brain is just a computer made out of meat and it doesn't do anything that can't be precisely simulated by a silicon computer.But I wonder if som
>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuationRegardless of bigger issues, this kind of statement reveals a deep misunderstanding.Problem type does not limit problem complexity. Nor does problem type limit solution complexit