657,607 links later: the web's memory is worse than you think

5 min read 1 source clear_take
├── "Link rot is decaying faster than the commonly-cited ~10-year half-life suggests"
│  └── tdx (0.mk) (0.mk blog) → read

After crawling 657,607 outbound links across blogs, wikis, and forums, the author found that pre-2015 links fail at rates well above 50% — significantly steeper than the popular ~10-year half-life figure. The scale and cross-referencing against the Wayback Machine give the finding more weight than the anecdotal estimates that usually circulate.

├── "The real danger isn't dead links — it's 'silent' failures that look alive but point to wrong or misleading content"
│  └── tdx (0.mk) (0.mk blog) → read

The author argues the failure surface has shifted: instead of loud 404s and DNS errors, today's rot is dominated by parked domains, squatter landers, and generic CMS installs that return HTTP 200 with plausible-looking content. A machine reader — or a casual human — will happily treat these as ground truth, making the modern failure mode worse than a clean broken link.

├── "Content never captured by any archive is the truly irrecoverable loss"
│  └── tdx (0.mk) (0.mk blog) → read

By cross-referencing dead links against the Wayback Machine, the author isolates the subset of vanished pages that no archive ever grabbed. This bucket — not the merely-offline-but-archived one — represents the genuinely permanent hole in the historical record and is what deserves the most concern.

└── "Anyone who treated URLs as stable pointers over the last decade now has a data-integrity problem"
  └── top10.dev editorial (top10.dev) → read below

The editorial frames the 0.mk numbers as putting a hard floor under a problem that's been widely acknowledged but never budgeted for. Any system built in the last ten years that treats a URL as a stable fact-pointer — citations, footnotes, knowledge bases, training corpora — now has a nontrivial fraction of pointers aimed at nothing, or at something plausibly wrong.

What happened

A developer running the blog at 0.mk published the results of a link-rot survey that's unusually big for a personal project: 657,607 outbound links pulled from a mixed corpus of blogs, wikis, forum posts, and archived pages, then checked one by one for whether they still resolve to anything resembling their original content. The methodology goes beyond the usual 'does it 200?' check — the crawler also flags soft-404s (pages that return 200 but say 'not found'), parked-domain landers, and redirects that land somewhere semantically unrelated to the anchor text.

The headline finding: link survival decays roughly with age, and the decay is steeper than the naive 'half-life of a link is ~10 years' figure that gets passed around. Links from before 2015 fail at rates well above 50% in the sample, and a meaningful chunk of the 'still alive' bucket is actually domain squatters, generic CMS installs, or archive-org placeholders that a machine reader would happily treat as ground truth.

The author cross-references against the Wayback Machine to distinguish 'gone from the live web but still recoverable' from 'gone, period.' That second category — content that no archive ever grabbed — is the one worth losing sleep over, because it's the part of the historical record that's genuinely, permanently missing.

Why it matters

Link rot is one of those problems everyone nods about and no one budgets for. The 0.mk numbers put a floor under the intuition: if you built anything in the last decade that treats a URL as a stable pointer to a fact, a nontrivial fraction of your pointers now point at nothing, or worse, at something plausible but wrong.

The 'worse than nothing' case is what's changed in the last five years. A dead link used to fail loud — DNS error, connection refused, a 404 page you could detect with a regex. Today, the failure surface is dominated by pages that return HTTP 200 with generic 'domain for sale' HTML, WordPress installs on expired domains that serve casino spam, or CDN-level catch-alls that redirect every path on a domain to the root. A naive link-checker sees 'up'; a human sees 'gone'; an LLM ingesting the page as retrieval context sees 'authoritative-looking content about online poker.'

The RAG angle is the one worth pulling on. Every retrieval-augmented pipeline that fetches a URL at query time — the 'grounded search' pattern that half the AI product teams shipped in the last 18 months — inherits this problem directly. If your citation store is a year old and you re-fetch on demand, you're one expired domain away from your assistant confidently citing a Kazakh gambling site as the source for a claim about the Linux kernel. The mitigation isn't 'check the link is up' — it's 'snapshot the content at ingest and treat the URL as a locator, not a source of truth.'

Academic and legal citation practice has been quietly moving this way for a while. Harvard's Perma.cc, the Internet Archive's Save Page Now, and the archive.today mirror all exist because law reviews figured out around 2013 that half the URLs in their footnotes were dead within a few years. The software industry is roughly a decade behind, and it shows: MDN, Stack Overflow, and GitHub README links that pointed to authoritative sources in 2018 are now a coin flip.

There's also a survivorship-bias problem in how we reason about the 'old web.' The parts of the pre-2015 web that we still see — the popular blog posts, the influential essays, the linked-to-a-thousand-times tutorials — are the parts that got mirrored, archived, and re-hosted. The long tail of technical writing, mailing-list threads, personal wikis, and small-forum expertise that made the early web valuable is disproportionately in the failed-link column. What we remember as 'the old web' is a heavily filtered signal.

What this means for your stack

If you ship anything that stores URLs, three concrete moves are worth the engineering time.

Snapshot at ingest, not at read. Whether you're building a bookmarking tool, a RAG index, a docs site with external references, or an email newsletter with source links, capture the content the moment you save the URL. The cheap version is a Wayback Machine `save` API call; the robust version is your own S3 bucket with the HTML, a screenshot, and a content hash. Storing the URL alone is a bug that hasn't shipped yet.

Distinguish 'link resolves' from 'link is still the same page.' A HEAD request that returns 200 tells you almost nothing in 2026. If the content matters, hash the visible text at ingest and periodically re-hash; a delta above some threshold is a signal to either re-verify or degrade the citation. Github's own dependency graph does a version of this for package manifests, and the pattern generalizes.

Assume outbound links in older content are rotten and treat them as such in the UI. If you run a blog, docs site, or knowledge base with a multi-year archive, run the 0.mk-style check on your own outbound corpus. The results will be worse than you expect. Options: strip dead links, replace with Wayback links inline (the `web.archive.org/web/*/` prefix trick), or annotate 'this link was verified on ' so readers know what they're getting. Silent rot erodes trust faster than a visible disclaimer.

For teams building on top of LLMs specifically: your evaluation harness needs a rot-check step. If your model's answers cite URLs, a fraction of those citations are already wrong, and that fraction grows monotonically with time. Bake the check into your regression suite the same way you'd bake in a check for hallucinated function signatures.

Looking ahead

The pessimistic reading of the 0.mk numbers is that the web is a lossy medium and we should stop pretending otherwise. The more useful reading is that the tools to work around it — content-addressed storage, cheap archival APIs, hash-based drift detection — are already sitting on the shelf, and the cost of adopting them is a rounding error compared to the cost of building a product on citations that quietly turn into casino spam. The teams that treat URLs as pointers to be snapshotted, not sources to be trusted, will spend the next decade being right about things the rest of us are confidently wrong about.

Hacker News 215 pts 197 comments

Where did the old web go? We followed 657,607 links to find out

→ read on Hacker News

// share this

// get daily digest

Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.