An estate agent’s brochure is a wonderful thing. “Four bedrooms,” it says. “Conservatory. Sun room.” You arrive and find three bedrooms, a patch where the conservatory used to be, and a sun room that somebody turned into a downstairs loo in 2019.
Nobody lied, exactly. The house changed. The brochure didn’t.
Colm, a chartered surveyor, has taken up a hobby. He reads websites the way he reads a property brochure: the public ones, politely, using nothing but what any visitor can see. And he has noticed that websites do exactly what the house did.
The true bit
I crawled 26,960 URLs across 41 small web agencies. The most interesting finding wasn’t broken metadata. It was that 49% of the sites were publicly contradicting themselves: their sitemap said a URL belonged on the site, while that same URL redirected elsewhere, was marked noindex, was blocked, or declared a different URL as canonical.
That matters because a sitemap is only one signal. Google’s guidance treats listing a URL in a sitemap as one hint about the preferred address, while redirects and canonical tags are separate signals about it.
So this isn’t simply an SEO mistake. It is drift between the intended state of a website and its current state.
Websites decay
Websites rarely break all at once. A page gets renamed. A CMS changes. A redirect is added. A campaign page disappears. A sitemap carries on listing an old URL. A canonical tag is changed somewhere else. Each change is small. Eventually, different parts of the site describe different versions of reality.
It isn’t only my sample. Pew Research examined close to a million webpages and found that a quarter of the pages that existed at some point between 2013 and 2023 were no longer accessible by October 2023. For pages from 2013 the figure was 38%. Even 8% of the pages from 2023 itself had already gone.
Why an old website is useful to an investigator
This is where the Internet Archive’s Wayback Machine gets interesting. Take a sitemap inconsistency today and compare it with an archived version of the page, an old navigation structure, previous service names, historic URLs, or earlier staff and contact details.
Suddenly a broken redirect isn’t just a broken redirect. It can be evidence of a migration. A stale sitemap entry can expose a previous site structure. A canonical mismatch can point towards pages that were consolidated or replaced. In other words, technical debris can tell you something about the history of an organisation.
The useful OSINT lesson
OSINT is often portrayed as finding obscure information. But some of the best signals are mundane: sitemaps, robots.txt, HTTP responses, DNS records, cached pages and archived websites. The value comes from comparing them. A current website tells you what an organisation looks like today. The archive tells you what it looked like before. The discrepancies tell you what changed.
The same trail is visible to anyone, so it is worth tidying your own.
The moral of the story
Digital decay doesn’t just create broken links.
It leaves a trail.
References
- The People Hacker, Agency websites 2026: research and methodology.
- Pew Research Center, When online content disappears (May 2024).
- Google Search Central, How to specify a canonical URL with rel=”canonical” and other methods.
- Internet Archive, Wayback Machine.