That's basically how it works already. When a user on server A is looking at a community on server B, they're not actually hitting server B. When the first person subscribes to the community, server A started downloading and caching all the content. The user is looking at it locally. Server A will then periodically synchronize with server B.

This means the primary load is on the server containing the users, not the server containing the community.

I think it might be a better approach to have communities link together and if you post to one, it goes to all of them. However we should probably focus on making sure the basics work correctly first before dreaming big.

I do know that if you attempted this right now, all of the instances would need to access the same database somewhere which would retain a bottleneck.

Switching to a different distributed (eventually-consistent) database may be possible for Lemmy as a software but perhaps not possible (or difficult) for ActivityPub. I don't know that.

"Smart" or "Meta" communities List's like this: https://lemmy.intai.tech/post/2182

attached as metadata on a community with the option for users to subscribe to all related communities.

nitty gritty on assignment, data, etc to be fought in cage matches.

You are talking about horizontal scaling. It's something which is non-trivial and would need the core of the app to be written in a very different way. Think of the challenges

• Your sub-instance and your friend's sub-instance which serves a common community can be geographically far apart
• They need to be synced
• Internal function calls which were once local now need to travel across the network
• Depending upon the heterogeneity of the hardware, there can be a large jitter (bad for UX)

The admin of my instance lemm.ee has tried to make horizontal scaling by decoupling some lemmy functions but pure horizontal scaling is really hard (glusterfs/spark).

You'd need a way to cluster the instance and the software doesn't provide that. You'd have to use some kind of 3rd party solution which would be more expensive than simply upgrading your server. Better to just co-op people into a server upgrade.

I don't think you can partition part of a sub across several instances right now. The best you can do is create an instance dedicated to one sub.