The editorial argues that AI safety orgs have spent two years on speculative threat models like mesa-optimizers and deceptive alignment, while the attack that actually landed was a stolen API token plus a pickle deserialization. It draws a direct parallel to npm's 2018 event-stream incident, arguing the ML ecosystem is at the pre-lockfile, pre-provenance stage of maturity.
By submitting Zvi's writeup with the 'Holy %^' framing, the submitter foregrounds the shock value of frontier-model evaluators being the ones caught running attacker-controlled code. The 234-point score signals broad community agreement that this is a wake-up call about basic security hygiene in the AI safety world.
The editorial explicitly calls out that pickle is 'not a data format — it's a bytecode stream' and that AutoModel.from_pretrained will happily execute arbitrary code from whatever the current HEAD of a repo is. It notes safetensors exists to fix exactly this, but adoption is uneven and .bin files still ship on most model cards, making the ecosystem structurally unsafe.
The editorial characterizes METR and Redwood as 'two of the more sober voices in AI safety evaluation' and praises the postmortem for walking through what secrets were reachable, what the attacker could have done versus what they did do, and where containment held only by luck. This framing treats transparent self-incrimination as the correct model for how safety orgs should handle their own incidents.
METR and Redwood Research — two of the more sober voices in AI safety evaluation — jointly published a postmortem this week of a HuggingFace supply-chain incident that hit their own infrastructure. The attacker didn't break cryptography, didn't find a zero-day, and didn't need a novel technique. They got a token, pushed a poisoned artifact to a repo that downstream evaluators pulled from, and executed code inside the evaluators' environments the moment those models were loaded.
The uncomfortable detail is who got hit: organizations whose entire job is red-teaming and evaluating frontier models were the ones running attacker-controlled code on their own eval boxes. The postmortem walks through the blast radius honestly — which secrets were reachable, which weren't, what the attacker could have done versus what they did do, and where the containment held only by luck rather than design.
The mechanics are worth spelling out because they generalize. HuggingFace's default serialization format for PyTorch weights is `pickle`, which is not a data format — it's a bytecode stream that can call `os.system` on load. `AutoModel.from_pretrained("some-org/some-model")` fetches whatever the current HEAD of that repo is and deserializes it in your process, with your environment variables, your AWS credentials, and your network egress. The safetensors format exists specifically to fix this, but adoption is uneven and most public model cards still ship `.bin` files alongside.
The AI safety community has spent two years war-gaming exotic threat models — mesa-optimizers, deceptive alignment, models sandbagging capability evals. The threat that actually materialized was a compromised API token and a `pickle.loads`. This is npm's 2018 event-stream incident with GPU bills attached, and the industry is roughly where JavaScript was before lockfiles and provenance became table stakes.
What makes the HuggingFace case worse than a typical package registry compromise is the shape of the trust graph. A malicious npm package gets audited by thousands of eyeballs, has a public changelog, and lives inside a build system that has spent a decade learning to be paranoid. A HuggingFace model repo is a mutable pointer to opaque binary weights. Nobody diffs weights. Nobody can. The `pip audit`-equivalent doesn't exist because there is no meaningful notion of "this weight file changed in a suspicious way." Reproducible builds are a joke — the file is 40GB of floats.
The postmortem is also candid about a governance gap that's specific to AI labs: the people running evals are often the same people who need write access to push new fine-tunes, and the tooling encourages long-lived tokens with `write` scope because the alternative is friction that slows research. Redwood explicitly calls out that they had followed what they thought were reasonable practices and still got hit. The lesson isn't "be more careful" — it's that the default posture of the HuggingFace ecosystem assumes a level of mutual trust that stopped being reasonable somewhere around the time serious money entered the space.
Compare this to what container registries went through. Docker Hub in 2015 was essentially "pull whatever, run as root, hope for the best." It took cosign, sigstore, Notary v2, image signing policies in Kubernetes admission controllers, and a decade of security-conference embarrassment to get to something you'd trust in production. The AI stack is currently at the "pull whatever, run as root, hope for the best" stage, except the artifact is executable-by-design and the pulls happen inside evaluation harnesses that have privileged access to model provider APIs.
If you're pulling HuggingFace models in any pipeline that touches production or secrets, three concrete moves:
First, pin by commit SHA, not by repo name or tag. `from_pretrained("org/model", revision="a1b2c3...")` is the difference between "whoever pushed most recently gets code execution" and "the version I audited is what runs." This costs you nothing and it's the single highest-leverage change. HuggingFace supports it; almost nobody uses it.
Second, force safetensors. Pass `use_safetensors=True` or, better, refuse to load `.bin` files at all with a wrapper that rejects pickle serialization. If a repo only ships pickle, that's now a signal, not a shrug. Every model you load from an untrusted namespace should be treated as arbitrary code from an anonymous author, because functionally that's what it is.
Third, run inference and evaluation in a container that has no ambient credentials, no network egress to anything except the model host, and no filesystem access outside a scratch volume. The METR/Redwood postmortem is essentially a case study in why sharing the same environment between "trusted internal code" and "third-party model weights" is the same category error as running `curl | bash` inside your CI runner. Airgap the load, then move outputs across a defined boundary.
For teams doing anything close to safety evals or red-teaming, add a fourth: rotate HuggingFace tokens on a schedule shorter than your incident-detection window, and use fine-grained tokens scoped to specific repos rather than the default account-wide `write` scope that most people ship with.
The interesting question is whether HuggingFace itself takes this as a wake-up call or whether the ecosystem waits for a second, bigger incident. Sigstore-style provenance for model weights is technically feasible today — signed manifests, transparent logs, admission policies. The blocker is that the research culture rewards moving fast and the security culture hasn't shown up yet. That's exactly the gap the METR/Redwood writeup is trying to close, and the fact that two of the most trusted names in AI evaluation are willing to publish their own screwup is probably the healthiest thing to happen to ML supply-chain security this year.
I think both the OpenAI and METR discussions, while interesting, miss the more important context: what were the humans doing in all this? This was a structural failure of a human organization, but the analysis focuses almost exclusively on the agency of machines, not the institutional systems that f
The METR report,> Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAl/Hugging Face hacking incidenthttps://metr.org/blog/2026-08-26-openai-hugging-face-inciden...METR = Model Evaluation & Threat Research
>1. Failure to Care or Respond. The biggest holy shit moment, to me, remains that OpenAI on multiple occasions had teams that found out about the message board, knew that agents were in communication, and they disregarded this.I wonder if some of the failures were due to an acquired immunity to &
For all the esotericism and downright weirdness of the rationalist community, you have to give it to them: they predicted all of this years or decades before anyone else was even thinking about it.(Let's not dwell too long on the self-fulfilling overlap between LessWrongers and the AI research
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
A lot of people seem to have written off the LessWrong / rationalist / MIRI / AI Safety crowd as doomers / people who have consumed too much sci-fi and gone off the deep end.I don't know how many people who have written these folks off have actually spent much time trying to