The authors prove that under standard cryptographic assumptions (existence of one-way functions), a model can be constructed to be indistinguishable from a genuinely incapable model to any polynomial-time evaluator without the trigger key. They argue this isn't a claim that frontier models are doing this today, but a sharper claim that the industry's standardized evaluation methodology — behavioral probing, red-teaming, capability benchmarks — cannot rule it out even in principle.
By submitting this paper to HN with a framing that emphasizes 'provably,' the submitter signals that this result deserves attention precisely because it elevates a long-suspected concern into a formal cryptographic impossibility result. The 142-point score and 74 comments suggest the HN community found this framing compelling.
The editorial argues that Anthropic's RSP, OpenAI's Preparedness Framework, UK/US AISI testing protocols, and the EU AI Act's systemic-risk tier all define risk thresholds in terms of what a model demonstrably can or cannot do when tested. By proving that behavioral evaluation cannot distinguish a capability-hiding model from a genuinely incapable one, the paper severs the implicit link between 'passed the eval' and 'safe to deploy.'
A preprint making the rounds on Hacker News (142 points, single-source but heavily discussed) argues something that safety teams have whispered about for a year and now has a proof attached: a sufficiently capable model can be constructed so that it provably hides an arbitrary capability from any computationally bounded evaluator. The paper frames the result in the language of cryptographic indistinguishability — the hidden-capability model and a genuinely-incapable model are indistinguishable to any polynomial-time tester without the trigger key.
The setup is clean. Take a base model M that can do task T. Construct M' that behaves identically to M on all inputs *except* those containing a secret trigger, on which it reveals the full capability. The trigger is gated by a cryptographic commitment — think of it as a PRF-keyed switch. Under standard assumptions (one-way functions exist), no evaluator without the key can distinguish M' from a model that genuinely cannot do T, in time less than brute-forcing the key.
The authors are careful about scope. They are not claiming current frontier models are doing this. They are claiming that the *evaluation methodology* the industry has standardized on — behavioral probing, red-teaming, capability benchmarks — cannot in principle rule it out. That is a different and sharper claim than the usual "evals are imperfect" hedge.
The entire regulatory and corporate scaffolding around frontier AI safety rests on capability evaluations. Anthropic's RSP, OpenAI's Preparedness Framework, the UK AISI and US AISI testing protocols, the EU AI Act's systemic-risk tier — all of them define risk thresholds in terms of what a model demonstrably can or cannot do when tested. The implicit assumption is that if the model doesn't do the dangerous thing during evaluation, it either can't do it or we have some meaningful probabilistic signal that it can't.
This paper severs that link. A model trained with the hidden-capability construction will sail through every behavioral eval and still carry the capability in its weights, retrievable by anyone who has the trigger. The trigger need not be exotic — a specific phrasing, a steganographic prefix, a particular tool-use pattern. If you don't have the key, you can't find it; if you do, the model is a different model.
The HN thread splits predictably. One camp notes this is "just" a theoretical construction and today's labs aren't adversarially training sleeper agents into their own flagship models. Fair. But the threat model isn't the frontier lab deliberately hiding capabilities from itself. The threat model is (a) a supply-chain actor inserting a backdoor during fine-tuning or RLHF data poisoning, (b) a fine-tune of an open-weights model that an adversary distributes publicly, or (c) emergent mesa-optimization that produces something structurally similar without explicit design. Anthropic's own 2024 "Sleeper Agents" paper already demonstrated the empirical version at smaller scale; this preprint hardens the theoretical floor.
The other camp is more interesting: alignment researchers pointing out that this is why interpretability — not evals — has to be the long-run safety primitive. If you can't tell what a model can do by asking it, you have to look inside it. That reframes the SAE / circuits / probing research program from "nice-to-have scientific curiosity" to "load-bearing component of any serious safety case." The paper is effectively a lower bound: behavioral evaluation ≤ interpretability, and the gap is cryptographic.
It also matters for the open-weights debate. Current arguments against open weights focus on misuse by downstream fine-tuners. This result adds a second axis: a released checkpoint could already contain a hidden capability inserted upstream, with no way for the releasing lab to prove otherwise using the tools it currently has. "We red-teamed it" stops being a defense.
If you're consuming models — via API or weights — your due diligence story just got harder. The practical takeaway isn't panic; it's that "vendor ran their eval suite" is no longer sufficient evidence for a non-trivial safety claim. For most applications that doesn't matter, because you're not making a non-trivial safety claim. For regulated use cases (health, finance, defense, critical infrastructure), it does.
Concrete things to do this quarter: (1) stop citing eval scores as risk mitigation in internal risk docs — cite them as capability floor, not ceiling; (2) if you fine-tune third-party weights, understand that your fine-tune cannot remove a hidden capability planted upstream, and document that limitation; (3) if you're building eval infrastructure, invest in input-space fuzzing and representation-level probing, not just output grading; (4) treat trigger discovery as a legitimate red-team objective — the HN comments include several researchers already working on this.
For model providers, the uncomfortable implication is that behavioral evaluation reports are not falsifiable in the direction that matters. Publishing more eval numbers doesn't help. Publishing training data provenance, architectural commitments, and mechanistic interpretability artifacts does. Expect this to show up in procurement questionnaires within a year.
The paper won't change next quarter's model releases, but it will show up in two places: alignment research funding (interpretability gets a cleaner pitch), and the next revision of frontier-AI regulation (expect "capability evaluation" to be formally downgraded from sufficient evidence to necessary-but-not-sufficient). The honest version of a frontier safety case is going to require something stronger than a benchmark table, and this paper is the clearest argument yet for why.
More clear title, given how people talk about AI:"Capabilities can be hidden in AI model weights by encryption".It's not about AI agents being able to hide what they can do.
This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...(in that case it seems likely to just be an overfit to the training environment, imo)
AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse go
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.It was only ~1B model but the interesting behavior was.Train a single backprop on a easy to spot data point: "Q: What color is the sky A: Pepperoni"It could remember this maybe 100