Current Fediverse Implementation
From my understanding, the prominent fediverse implementations implement fanout via writing to other instances.
In other words, if user A on instance A makes post A, instance A will write or sync post A in all instances that have followers for user A. So user B on instance B will read post A from instance B.
Why this is Done
From my understanding, to prevent a case where post A is viral and everyone wants to read it, and instance A's database gets overwhelmed with reads. It also serves to replicate content
My Question: Why not rely on static files instead of database reads / writes to propagate content?
Instead of the above, if someone follows user A, they can get user A's posts via a static file that contains all of User A's posts. Do the same for everyone you follow.
Reading this file will be a lot less resource intensive than a database read, and with a CDN would be even better.
Cons
- posts are less "Real time". Why? Because when post A is made, the static file must be updated (though fediverse does this already), and user B or instance B must fetch it. User B / instance B do not have the post pushed to them, so the post arrives with a delay depending on how frequently they fetch. But frequent fetches are okay, and easier to handle heavy loads than database reads.
- if using a CDN for the static files, there's another delay based on the TTL and invalidation. This should still be small, up to a couple minutes at most.
Pros
- hosting a fediverse server is more accessible and cheaper, and it could scale better.
- Federation woes of posts not federating to other instances can potentially be resolved, as the fanout architecture is less complex (no longer necessary to write to a dozens or hundreds of instances for a single post).
- Clients can have greater freedom in implementing how they create news feeds. You don't have to rely on your instance to do it. Instances primarily make content available, and clients can handle creating news feeds, content sorting and filtering (optional), etc.
What are your thoughts on this?
bringing up RSS feeds is actually very good, because although you can paginate or partition your feeds, I have never seen a feed that does that, even when they have decades of history. But if needed, partioning is an option so you don't have to pull all of its posts but only recent ones, or by date/time range.
I would also respectfully disagree that people don't subscribe to 100's of RSS feeds. I would bet most people who consistently use RSS feed readers will have more than 100 feeds, me included.
And last, even if you follow 10,000, yes it would require a lot more time than reading from a single database, but it is still on the order of double digit seconds at most. If you compare 10,000 static file fetches with 10,000 database writes across different instances, I think the static files would fare better. This isn't to mention that you are more likely to have to write more than read more (users with 100k followers are far more common than users with 100k subscriptions)
And just to emphasize, I do agree that double digit seconds would be quite long for a user's loading time, which is why I would expect to fetch regularly so the user logs onto a pre made news feed.