Werdmuller argues that while the Climate.gov website was dismantled — staff fired, editorial shut down, content pulled — the underlying datasets survived because they had been mirrored across universities, research groups, and the Internet Archive long before the takedown. His central claim is that the data was never actually stored in only one place, so the website's death was a presentation-layer failure, not a data loss event.
Werdmuller emphasizes that projects like the Public Environmental Data Partners, Data Rescue Project, and End of Term Web Archive had been rehearsing this exact scenario since 2016. When the takedown finally came, the response was routine — cron jobs fired, rsync ran, checksums matched — because the mirrors were already warm, showing that resilience is built through boring, sustained infrastructure work rather than dramatic rescue efforts.
The editorial draws a contrast between Climate.gov's recoverable data and what happens when proprietary datasets are pulled — vendor API sunsets, scraping target changes, or private provider acquisitions leave no mirrors and no recourse. The implication is that open data with distributed replication is architecturally superior for long-term availability, and modern SaaS products routinely make the mistake of treating single-vendor dependencies as if they were durable substrate.
Climate.gov — NOAA's public-facing hub for climate education, data visualizations, and decades of curated research — was effectively dismantled this year. Staff were let go, the editorial operation was shut down, and much of the site's content was pulled or degraded. For a certain kind of reader this landed as a political story. For engineers who actually use this data, it landed as an availability incident with no ETA on recovery.
And yet the data itself did not disappear. As Ben Werdmuller documented in the post that hit the Hacker News front page (483 points), the underlying datasets — temperature records, sea-level series, NOAA's Climate Data Online exports, GHCN, ERSST, the whole familiar alphabet soup — were already mirrored across universities, research groups, the Internet Archive, and a growing constellation of independent replicas that spun up the moment the takedown risk became credible. The website died. The data did not, because it was never actually stored in only one place.
Projects like the Public Environmental Data Partners, Data Rescue Project, and the End of Term Web Archive had been quietly rehearsing this scenario since 2016. When the actual takedown arrived, the response was not heroic — it was rote. Cron jobs fired. rsync ran. Checksums matched. The mirrors were already warm.
There's a tempting narrative here about civic heroism, but the more useful frame for a working engineer is architectural. Climate.gov was a single point of failure sitting on top of data that had been engineered, deliberately, to not have single points of failure. The site is a presentation layer. The data is the substrate. Confusing the two is the mistake — and it's a mistake a lot of modern SaaS products make with their own dependencies every day.
Compare this to what happens when a proprietary dataset gets pulled. When a vendor sunsets an API, or a scraping target rewrites their terms, or a private data provider gets acquired, the downstream consequences are immediate and often unrecoverable. There is no mirror. There is no checksum. There is a support ticket and a roadmap slide. The reason the climate data survived is not that climate scientists are morally superior to SaaS PMs — it's that NOAA published in open, documented, versioned formats with permissive terms, and a community had spent a decade treating those files as canonical rather than treating the website as canonical.
The HN thread on Werdmuller's post is worth reading in full, but the recurring theme from practitioners is telling: several commenters mentioned they'd been pulling NOAA data into their own S3 buckets for years, not out of political foresight, but because government websites are historically flaky and their pipelines couldn't tolerate a 500. The people best positioned for this moment were the ones who had already given up on trusting a .gov URL as a stable dependency. Resilience, it turns out, is often a byproduct of low expectations.
The policy conversation about whether this takedown was legal, appropriate, or reversible is important and outside our lane. The engineering conversation — the one that actually determines whether the next dataset survives — is about defaults. Open license by default. Bulk export by default. Content-addressed storage by default. Federation by default. Every one of those defaults is boring, well-understood, and available today, and every one of them is missing from the majority of the data infrastructure the modern web actually runs on.
If you ship a product that depends on an external dataset — government, academic, commercial, doesn't matter — the Climate.gov episode is a free tabletop exercise. Ask three questions. First: if the source disappeared tomorrow, do you have a complete local copy of what you actually use, not just what you cached last week? Second: is your copy verifiable — checksums, signatures, provenance metadata — so you can prove to a customer or auditor that the numbers didn't drift? Third: is your ingestion pipeline idempotent and scheduled, so a mirror is a byproduct of normal operation rather than a heroic side project?
Most teams fail question one. Almost everyone fails question two. The fix is not exotic. A nightly job that pulls the upstream, computes SHA-256 over the payload, writes to versioned S3 (or R2, or a local ZFS pool), and emits a Prometheus metric on drift will cover 90% of the risk for a week of engineer time. Add a manifest file with dataset version, source URL, retrieval timestamp, and license terms, and you've also solved a chunk of your compliance story for free.
The deeper takeaway is about how you evaluate any external dependency. "Is there an API" is the wrong first question. "Is there a bulk export I can hold in my own bucket, under a license that lets me hold it" is the right one. If the answer is no, you don't have a dependency — you have a rental agreement with a landlord who can change the locks. Every serious data-dependent product should have a written policy about which of its inputs are rentals and which are owned, and a plan to convert the important rentals into ownership before they become incidents.
The uncomfortable read on this story is that Climate.gov was easy mode. The datasets were already open, already bulk-exportable, already mirrored by people with institutional support. The next takedown — of a dataset that isn't as open, or a corpus that lives inside a proprietary training pipeline, or a public API that quietly becomes private — will not have a decade of rehearsal behind it. The response window for building the mirror is always before you need it, never during. The engineers who came out of this week looking prescient did nothing clever; they just took the boring advice seriously about ten years ago. That option is still on the table for the rest of us, for exactly as long as the current data is still reachable.
Partisan politics aside, frankly, anything data the government publishes like this should be public domain by virtue of it being published by the government.How can the government "for the people by the people" claim propriety/intellectual-property over anything?
What if government websites were distributed & archived as a default, from the beginning? Think IPFS as a first target for publication, "normal web" only as a mirror.Is it feasible?Should we push for this default?First obvious objection is that lots of government services need backend
A similar thing happened back in Australia years back.We had the Climate Commission, which did alot of great work. It was a government department and fully funded by the government.Then a conservative, climate denying government came to power and shut it down.Literally within 48 hours, they obtained
Blog post from a few weeks ago: https://www.climate.us/news-features/feed/climateus-launches... (https://news.ycombinator.com/item?id=48689182)
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.
I'm glad someone managed to save the data that we all payed for.My question is, how will this site stay relevant? The collection/analysis/monitoring of the current situation is as important as historic data. Turning current data into historical data takes significant resources.