The fake think tank was built for ChatGPT, not for humans

5 min read 1 source clear_take
├── "This is a template for LLM-targeted influence operations — the target is the model, not the human reader"
│  ├── Responsible Statecraft (Responsible Statecraft) → read

The original investigation frames the fake think tank as a deliberate attempt to dupe AI chatbots rather than sway human readers. It highlights invented expert bylines, templated bios, and a publication cadence no real DC policy shop could sustain — signals of an automated pipeline optimized for machine consumption.

│  └── top10.dev editorial (top10.dev) → read below

Argues the operation's tell is structural: dense, citation-friendly prose with declarative headlines and paragraph-closing source links — exactly what retrieval-augmented generation systems reward. The specific op almost doesn't matter because the template is now public and cheap to replicate.

├── "Retrieval-augmented AI has turned citations from a trust signal into an attack surface"
│  └── top10.dev editorial (top10.dev) → read below

Points out that every major consumer AI product — ChatGPT search, Perplexity, Gemini grounding, Claude's web tool — now pitches citations as proof of trustworthiness. But citations are a ranking signal adversaries can manufacture, inverting the security model that made grounded answers seem safer than raw generation.

└── "LLM-facing content farms escape the human bullshit detector that killed old SEO spam"
  ├── top10.dev editorial (top10.dev) → read below

Contrasts old SEO — where a human clicking through could smell a sketchy domain, ad-walled prose, or awkward translation — with LLM ingestion, which has none of those instincts. Content only needs to look structured and citable to a machine, lowering the bar for manufactured authority.

  └── @Hacker News discussion (Hacker News, 249 pts) → view

The 249-point thread's top comments focused on the mechanic rather than the politics, framing this as what an influence op looks like when the target user is a language model, not a voter. The community treated the technique — not the specific actor — as the newsworthy development.

What happened

Responsible Statecraft this week detailed what looks like a textbook LLM-poisoning operation: a shell organization presenting itself as an independent policy institute, publishing pro-Israel foreign policy content under invented expert bylines, with a website architecture that reads less like a think tank and more like a content farm optimized for machine consumption. The site's fingerprints — templated author bios, thin sourcing, aggressive cross-linking to other suspect domains, and a publication cadence no real DC shop could sustain — point to an automated pipeline rather than a research operation.

The tell isn't that the content is bad; it's that the content is dense, structured, and citation-friendly in exactly the way retrieval-augmented generation systems reward. Headlines are declarative. Paragraphs open with the claim and close with a source link. Named "experts" have LinkedIn-adjacent bios but no verifiable footprint at real institutions. The domain is old enough to have accrued some backlink equity but new enough that nobody in the actual policy world has heard of it.

Hacker News surfaced the story to 249 points within hours, and the top comments zeroed in on the mechanic rather than the politics: this is what an influence op looks like when the target user is a language model, not a voter. Whether the specific operation succeeds is almost beside the point. The template is now public, and it is cheap.

Why it matters

Every major consumer AI product now ships some form of retrieval — ChatGPT search, Perplexity, Gemini's grounding, Claude's web tool. The pitch to users is that citations make the answers trustworthy. The pitch to adversaries is that citations are a ranking signal you can manufacture.

The old SEO game optimized for a human clicking through a search results page. That human could smell a farm: sketchy domain, wall of ads, prose that reads like it was translated twice. LLM-facing content doesn't need to survive human scrutiny — it needs to survive a chunking pipeline, an embedding model, and a summarizer that has been explicitly trained to synthesize rather than doubt. Once the fake think tank's paragraph lands in a retrieval index, the model treats it as a peer of a Brookings report, weighted by whatever ranking heuristic the retriever uses. That heuristic is almost always some combination of semantic similarity and domain authority — both of which are gameable at the scale a state actor operates.

The defensive literature is thin. There is real work on prompt injection, on jailbreaks, on training-data provenance — but the middle layer, the live retrieval index, is largely trusted by default. Bing's index, Google's index, Common Crawl, the various web APIs Perplexity and its peers hit: none of them were built with an adversarial threat model where the adversary is specifically producing content for an LLM to ingest. Google spent twenty years hardening its ranking against SEO spam aimed at humans. The RAG stack is starting that fight from scratch, with less telemetry and higher stakes, because a poisoned answer feels authoritative in a way a poisoned SERP never did.

Community reaction has split roughly two ways. One camp treats this as a solved problem — "just add source-quality filters" — which underestimates how quickly the arms race compounds when the attacker can spin up a hundred plausible domains for the price of one. The other camp treats it as unsolvable and argues LLMs shouldn't cite live web content at all, which is a fine position that will lose to the product managers shipping citation features next quarter. The honest read is somewhere in between: this is a hard, ongoing problem that most AI companies do not yet staff for.

What this means for your stack

If you're building anything that pipes web content into a model — a support bot with a docs index, a research agent, an internal Q&A over a public knowledge base — you now have a supply-chain problem that looks a lot like npm's. Your retriever is running arbitrary text through a summarizer that treats structure as a proxy for authority, and you probably have no allowlist.

A few things worth doing this quarter. First, audit what your retrieval layer actually pulls from. If you're using a third-party web tool (Tavily, Serper, Brave, or a hosted RAG provider), find out what its source-quality signals are and whether they're tunable. Most default to "relevance," which is exactly the signal an operation like this optimizes against. Second, if you're indexing your own crawl, add domain-age, backlink-graph, and author-verification signals to your ranking — the same signals Google's spam team has used for a decade, ported into your vector store. Third, in the prompt itself, ask the model to name its sources and flag when a claim rests on a single low-authority domain. This doesn't stop the attack, but it turns a silent failure into a visible one your users can push back on.

For consumer products, the harder question is disclosure. When a user asks about the Middle East and your model cites a domain that turns out to be an influence op, what's your incident response? Do you have logging that lets you trace which retrieved chunks fed which answer to which user? Most teams don't, and they will wish they did the first time a journalist asks.

Looking ahead

This will get worse before it gets better, because the economics favor the attacker: one engineer with an LLM can produce a year of plausible think-tank output in a weekend, and the cost of a poisoned domain is a $12 registration fee. The defense has to be structural — provenance signals baked into retrievers, adversarial evals that specifically test for coordinated inauthentic corpora, and probably a slow migration toward citation graphs that weight verifiable institutional affiliation over raw domain authority. The AI labs that treat this as a first-class safety problem alongside jailbreaks and hallucinations will end up with a real moat. The ones that treat retrieval as a solved integration will keep shipping bots that confidently cite ghosts.

Hacker News 942 pts 556 comments

Israel creates fake think tank in likely attempt to dupe AI chatbots

→ read on Hacker News
2001zhaozhao · Hacker News

I suspect that this kind of tactic is going to be everywhere in a year or so. Entire fake personalities and organization websites on the Internet created just to push a narrative or to advertise a product, which completely drown out real information.At some point they may be indistinguishable from h

karim79 · Hacker News

Just to put this truth out here again because apparently a lot of erm "people" don't like it:Itamar Ben-Gvir, just recently, stated that he wants to kill 30-40 Palestinians every night. He said so, publicly.He said that on TV. On TV.So it is hilarious that Israel is trying to influenc

xbmcuser · Hacker News

This is nothing new Israel has been doing this for years. It's just that people have started waking up to their lies.

techteach00 · Hacker News

Foundation for Defense of Democracies is another Israeli think tank that poses as an American organization. If you see anything quoted from them realize it's fake propaganda in the service of a foreign country.

karim79 · Hacker News

Itamar Ben-Gvir, just recently, stated that he wants to kill 30-40 Palestinians every night. He said so, publicly.So it is hilarious that Israel is trying to influence chatbots when shit like that is out in the open.

// share this

// get daily digest

Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.