Research shows 25% of web pages posted between 2013 and 2023 have vanished. A few organisations are racing to save the echoes of the web, but new risks threaten their very existence.
It's possible, thanks to surviving fragments of papyrus, mosaics and wax tablets, to learn what Pompeiians ate for breakfast 2,000 years ago. Understand enough Medieval Latin, and you can learn how many livestock were reared at farms in Northumberland in 11th Century England – thanks to the Domesday Book, the oldest document held in the UK National Archives. Through letters and novels, the social lives of the Victorian era – and who they loved and hated – come into view.
But historians of the future may struggle to understand fully how we lived our lives in the early 21st Century. That's because of a potentially history-deleting combination of how we live our lives digitally – and a paucity of official efforts to archive the world's information as it's produced these days.
However, an informal group of organisations are pushing back against the forces of digital entropy – many of them operated by volunteers with little institutional support. None is more synonymous with the fight to save the web than the Internet Archive, an American non-profit based in San Francisco, started in 1996 as a passion project by internet pioneer Brewster Kahl. The organisation has embarked what may be the most ambitious digital archiving project of all time, gathering 866 billion web pages, 44 million books, 10.6 million videos of films and television programmes and more. Housed in a handful of data centres scattered across the world, the collections of the Internet Archive and a few similar groups are the only things standing in the way of digital oblivion.
I’m part of the problem, a tiny bit. For altruistic reasons - ok more like “I’m kinda weird, maybe this will make people on IRC like me more” reasons - I ran mspencer.net and hosted web pages for people for free. Ended up with web content for around 100 people, and they weren’t all just using it as a drop box. (Older than wikipedia.org by 199 days, woo!)
Hosted on ancient hardware, nothing even remotely approaching a modern security architecture, I eventually left it to run un-maintained until the IDE HDD died. More recently I got the data off of it. (Heads unstuck themselves while in a cardboard box for a decade? Dunno.) But I don’t know how to get everything back online in a safe way.
I’m a proper software engineer now, I can kinda see how work handles securely hosting web services. Now just throwing everything together on one box feels too lazy and insecure. But I can’t figure out a reasonable security architecture to use. I thought I had one, but I failed to account for VM jackpotting attacks. And it feels like it takes me a month to do what a competent ops person can do in a day.
But that’s a discussion for a different comment section.
If it's static content, nothing beats an AWS S3 bucket.
Last time I went snooping:
15 installs of phpbb, which would require work to put back online as their communities are of course gone. Remove spam, undo defacement, etc.
7 installs of Dormando’s Oekaki BBS Clone
5 installs of WonderCatStudio BBS
4 installs of OekakiPotato / RanmaGuy etc.
and several users who just used php to ‘include’ headers and table of contents page parts.
(Yes I was quite the weeb. Still am, but I was one too. :-) )
If this was my problem to solve, I would host it internally, as-is, on a virtual machine of your choice, then create a a static html mirror version from the public information and put that up on AWS S3 as a static website.
That does make a lot of sense.
I think I’m feeling embarrassed about not being a perfect ops person, while I was going to school for computer science. Like, part of me wants to create this unrealistic private cloud thing, like I’m going to pretend “I’m still around, where have you been? See your old password still works, and look at all the awesome stuff I can do now!”. I already have my 20+ year old passwd file imported into OpenLDAP / slapd and email is using that already.
It’s not realistic. I feel fondness for the internet of 20-25 years ago, but it’s not coming back. If people can log in with 20 year old passwords and upload web content, we both know what’s really going to happen.
I just feel like such a failure for letting it rot away. Really, any place that accepts submissions requires a live audience and staff to keep it moderated, and accepting new submissions is the only reason to even run original code. What you’re describing is probably the only sane way to do this.
Edit: although I do still feel that the world needs that sort of private cloud in a box. Sure Facebook has taken all the wind out of the sails of many private web hosting efforts - the “family nerd” no longer gets love and gratitude for offering to host forums and chat, they get “that’s stupid, I’ll just use Facebook” - but we still need the capability.
And an open security architecture to clone would help cover the daylight between “here’s a web app in a docker container” and an actual secure hosted instance of it. It would require more inconvenience than necessary for the substantial security benefits it would offer. (A better designed, more customized solution would help that, but one step at a time.) But that would give the average homelab user protection against future attacks that today would feel like wild “whoa who are you protecting against, the NSA?” paranoia.