@nutomic@lemmy.ml brought up https://github.com/LemmyNet/lemmy/issues/5871 , which would at least be a way for communities to explicitly link to each other, and we could possibly create a superset of Subscribed.

In 1.0 there is the Suggested filter, where admins have a preset / chosen list of curated communities they like, and a way to view all the posts from them.

I'd rather not do complicated algorithms to try to figure out interest / community adjacency based on user activity, as this could get really complicated and also probably show things people don't want to see.

I totally agree that content discovery could be better, but I'd like it to be explicitly chosen by the user, rather than generated. So I think the best way is still just to go to the communities page, and click subscribe on anything that might potentially interest you.

You can't have a recommendation algorithm on open-source software because it requires a lot of compute to calculate personalized recommendations for each user, which simply isn’t feasible for most instances. Instead, there should be an API endpoint that returns post metadata for the last week, allowing users to implement their own ranking algorithm via a userscript running on their own hardware.

I also believe there should be a more personalized "All" feed per instance. Each instance could surface different content tuned to the admins or to a subset of long-term users—something stable that doesn't change often but varies from server to server.

Mastodon has an algorithm in their official app. The Explore feed shows you posts that are popular among people you follow and interact with.

Loops has a For You Page algorithm.

There are algorithms on the fediverse. They are rare, but they do exist.

The problem with recommendation algorithms isn't just the power, it's the fact that it deprives us of a shared reality. It's one thing if we filter ourselves into a bubble but it's another if the site itself does it.

i want to see posts from communities that i already subscribed to, but because there's more than 1000 communities on the fediverse and i'm only subscribed to a small countable subset of them, i inevitably lose out on a lot of content. (The "all" feed sucks unfortunately). So how to solve this?

You don't use the Subscribed feed? I like Subscribed+Scaled

The Fediverse significantly lacks behind on the Content Discoverability technology.

https://quiblr.com/

https://quiblr.com/understanding_your_private_personalized_feed

There should be an open-source recommendation algorithm, though; I’m sure of it.

Problem is that the kind of algorithm you envision is technologically a black-box, not just by choice. It's a machine learning model. At best, you could make the training data and instructions public, but it would still be hard to reason why it makes certain decisions. Corporations traditionally try to eliminate biases by throwing as much data at it as possible, but that makes it even harder to reason about it.

I guess, maybe you could try to split the tasks. So, set up a list of e.g. 50 topics, such as sports, IT, politics etc.. Then use a small language model to decide into which categories each post fits. And then you could let the user decide the weights for the topics + weights for recency and vote count.
Or I guess, automatically decide the weights based on what the user upvotes and then make the weights transparent to each user.

But yeah, I don't think there's prior art in this respect, so would probably need lots of experimenting still.