118
How will lemmy instances survive if they get too big?
(lemmy.world)
A loosely moderated place to ask open-ended questions
If your post meets the following criteria, it's welcome here!
Looking for support?
Looking for a community?
~Icon~ ~by~ ~@Double_A@discuss.tchncs.de~
I’ve been wondering about this not so much from a performance perspective, but rather from a storage perspective. Assuming old posts and comments are never cleaned up, a handful of particularly active instances could easily crush federated smaller instances storage, right?
Not as big of an issue as you'd think. In 2015, the entirety of Reddit could be archived using ~2TB of storage space. Keep in mind that this is the size when saving everything as uncompressed JSON, so the actual DB size would have been even smaller than this!
You're much more likely to see this DDOS-like effect coming from ActivityPub traffic volume. High activity = Lots of messages = Lots of processing demand. As the Fediverse grows, the baseline processing power necessary just to keep a small instance afloat will steadily rise, but it's not yet clear how big of an issue this will actually be.
We already saw a pretty major rash of Lemmy instances getting overwhelmed back on July 1st, but a lot of that bottlenecking was caused by quirks specific to Lemmy (which can be ironed out) rather than overhead inherent to the ActivityPub protocol itself (which, to be fair, is still relatively heavy).
From my understanding a instance will only interact with posts and comments (store them) when a user on that instance does. So just federating with a older instance that has a large back catalogue does not mean it will flood the small instance.
Side note about images. They seem to only ever be saved locally on the server it was uploaded to. So custom emojis like will not be saved on for example lemmy.ml (here). Only on my self hosted instance. Perhaps if a post is really popular and I link a self hosted image like that, it might put strain on my host to deliver this image though. Worst case just my small instance will go down and the link will be stale.
Side-side note: I might want to make all emojis smaller so that they both fit in the text better and put less strain on my host.