I realized the irony of posting this on unpopular opinions hosted on lemmy.world, which is actually demonstrating the issue!
To preach to Romans, you gotta go to Rome.
I go to Mr Hero FOR DELICIOUS ROMANBURGER!!! FEED THE HUNGER!!!
would be cool to link to current federation issues so people would actually see what's happening and how vast the problem is. I'm sure someone must've already made a list
Good point, here is a post I made today, you can see I don't get the comments on my lemmy instance. https://lemmy.world/post/15702974 vs https://hackertalks.com/post/3065575
I also updated the main post with general links to the issue. My one example just demonstrates it's happening to other instances not just Australian instances, but the Australians have done the best documentation of it so far.
For that example, there may be something else going on. The time behind graph for Lemmy World- > Hacker Talks seems to only average about 90 seconds. Some of the comments that show up on LW were made hours ago. If it were just federation delay, they should have arrived already.
Assuming those are going straight to your Lemmy server and not to a proxy that buffers them? If you've got a federation proxy buffering those, then the graph would be off since it wouldn't show what's sitting queued in your buffer.
You still definitely have a point. Every couple of weeks, my instance seems to start lagging behind by several hours and then goes back to normal after a few days. Still not sure why unless there's just a huge uptick in activities being sent out from LW. I made a post about it yesterday, but it doesn't dive too deep into the technical details; basically just an info post to let people on my instance know I'm aware and trying to do what I can about it.
Good point. And thank you for the graph. It could be that the activity pub protocol is best effort, and if lemmyworld is overloaded, or just dropping packets, whatever activity was supposed to be sent at that moment gets lost forever. That would explain me divergence, and lack of eventual consistency
But I only have data for missing posts from lemmy world, and only some communities.
I don't know the actual cause of the issue, but I think it is definitely related to the size of the instance. So from an empirical perspective the advice is still promote communities on non-mega instances
There is a very simple explanation for this specific case: nobody on hackertalks.com is subscribed to !goodoffmychest@lemmy.world!
Most community related activities on Lemmy will only be sent to instances that have at least one subscriber for the community.
sshhh… it just works :)
cause shit just works, yknow?
I always read it like : shit just works.
Never thought about it as sshhh
Leave Lemmy.world for other reasons too.
I’m still irritated that when I signed up last summer, it was like: it doesn’t matter which instance you choose because we’re all connected.
It’s not true. Lemmy.world relies too much on banning communities and even entire instances. I am the one who should be entitled to make those decisions, not Lemmy.world.
Consider coming to SDF Chatter or Lemmy.today instead.
Remember when they allowed the piracy communities just to get rid of them again?
I was gone by then and smugly rubbing my hands together from across the room mwahaha
Just like libertarianism would soon lead to single party Government, then eventually to Democracy, most people just want a less shit version or Reddit.
This whole Federation thing is a good principal, but most people just wanted a less crappy version of Reddit.
Most people may have just wanted a less crap version of Reddit, but how you make Reddit less crap is that you spread out the moderation and also don’t grow power in admins. That’s what federation (mostly) does. I just think we need a way to transfer a user and also to load multiple profiles.
I also don’t understand why federation needs to be the way it is. I feel like instances could present a “read-only” version of themselves to defederated instances so that users could at least read the posts but not interact.
Updoot the zoot! Doesn't even make sense, and everyone still knew what I meant.
I used to be on feddit but it shit the bed and the admin hasn't been able to fix it for over a month.
Before that I was on lemmy.ml but I was banned for speaking truth to China.
The problem is how Lemmy itself was built.
Decentralization should be a background thing with hosts providing server space like you would get from a service like AWS and the front end being a single website with users not knowing on which server their content is hosted and backed up.
If the front end is a single website, then it can be taken down, and provides a central weak point.
Not really if in the background everything is divided over a bunch of servers and backed up by other servers
It's the only way to solve the centralization of users, take the option away from them and handle it in the background.
Let's say whoever is running the front end doesn't like a community and blocks it... How do we prevent that?
You don't have "somebody running the front end" though, it's all done by the people providing hosting services.
Think "crypto philosophy as a message board" but instead of having everyone sync the whole history you split all data randomly in a way that guarantees it is stored on three servers at all time.
Heck, you could also have multiple front ends if you wanted, all pulling and pushing data to the same servers and this way you could log in from any of them, the front end would only have an influence on UI/UX, in the background the data would always come from the same places and for this reason the front end dev wouldn't have the power to block communities.
Ok. So just like Lemmy but communities are spread using some hash table over multiple existing nodes?
Yep, divide everything so no one has real power and it's the users that decide what they want and don't want on their feed, allow the hosts to decide if they want to host NSFW content or not (and the users can make the same choice), make it so the users don't have to decide what instance they register with, their credentials are just stored in the database.
For the front end you then have two ways to deal with it, a single one and the hosts "vote" on how to deal with it (crypto style) or the hosts are the database that anyone can access and it allows anyone to create a front end...
Your second scenario where the hosts decide and the users choose the hosts is what Lemmy is doing now
Not really though, with Lemmy the host and front end are the same. You access Lemmy from hacker talks, I access it from shitjustworks, both our respective instances host our respective data and provide us a UI to access their instance and could decide to only give us access to the content that they host as some instances already do.
What I'm talking about is front end devs not having to host any of the content themselves, just accessing the database hosted by others and showing that info in the UI they developed and pushing changes to the database when the users sign up or post comments.
Front end doesn't have control over where the info is stored and don't store anything locally, back end doesn't have control over who's pulling and pushing data, they can just choose to filter out NSFW content from their own servers but it just means it won't pick them to host that content and will instead pick servers that don't mind hosting it.
In this way the hosting is the same as any hosting services but completely decentralized and the data is open to all and no host can wipe it because of the backups (contrary to Lemmy where if an instance disappears all the content it hosted is gone).
So the data would be stored in something like IPFS? Or ceph?
Ceph has issues because the The metadata nodes needed for disc admission are central points of control
IPFS could work, but the cost of hosting go up exponentially. But yeah you could build Lemmy on top of IPFS
Like raid 6 or higher. Data is distributed with included redundancy to make up for nodes dropping off.
But then community admins would need to backup their communities as raid is not backup.
I just don't know how that would work with the data.. restoration distribution across new nodes etc etc. but then .. I'm not a dev.
Raid doesn't work in this context. Because we're assuming we have antagonistic peers. So Central control of any element, gives away control of the whole system.
In a redundant array of inexpensive disks, there's the assumption that there's bunificent administrator organizing everything.
I get that.. sort off.
In Raid the admin supplies the disks, creates the pools and the raid platform does the rest is this really different?
In the analogy it would be an admin starts a pool of 1, other admins join their node into the pool and the system handles distributing content across the nodes in the pool. No raid level selection as the system aims for optimal redundancy.
I just expect this setup to run into similar issues surrounding equitable data and load distribution, as not all nodes will be equal in power, storage capacity, bandwidth etc etc. something that actual Raid arrays should not have..
But it's cool to think about.
As a DevOps Architect, let me make it simple:
With a single front-end, you have a bottleneck. If you have one domain (website) that everybody goes to to get to the front-end, that means that domain is the single point of failure.
In my line of work, we use load balancers and sub-domains to divide the work and provide resilience (High Availability), but at the end of the day, if the DNS for that site goes down, we're down.
Also, as Jet mentioned, ~~whomever~~ whoever controls the domain (website) controls the content. You can't have multiple groups controlling a single domain. Whomever buys it controls it. If they don't like content, they could easily block access to it.
I'm oversimplifying the inner workings, so if you want more details, let me know.
EDIT: subtext called me out on my crap English. Have nobody to blame but myself. English is my first language.
In this case the solution, as I mentioned in other comments, is to make the back end the decentralized database that's accessible to anyone so the people developing a front end don't host the data and you can use any of the available front ends to connect to your account as it's not attached to any specific front end (your info is in the database).
Front end devs would be competing to provide the best UI/UX, but in the end everyone would have access to the same data and front end devs couldn't get in the way of the data or if they did then people could just go to another website without losing anything.
I just want to let you know that “whom” is only ever used as an object. In your sentences, I think you should have used “whoever”.
The easiest way to remember which you should use is to think about the difference between s/he==who and her/him==whom.
She gave the ball to him Who gave the ball to whom
She controls the domain Who controls the domain
The domain is controlled by him The domain is controlled by whom
Updated the comment with your recommendation. Yeah. I suck at writing.
Decentralization should be a background thing with hosts providing server space like you would get from a service like AWS and the front end being a single website with users not knowing on which server their content is hosted and backed up.
You could potentially run into issues with data storage reliability:
I understand that these things could still happen, to a similar extent, with the current model of Lemmy, but they are less likely to occur, given that you can choose which instance to join. These are all not unsolvable issues, but this is not a simple "better" alternative — it's more complicated than that.
All this being said, there is a service that I have heard a little bit about that is sort of similiar to what you appear to be looking for called Nostr.
World was the first one where I got an account quickly without some weird application process. And now I can't easily migrate.
It's also easy to sign up at lemmynsfw
Every time I add a new instance I gotta go in and block all the communities I don't like all over again
That's why I made an account on sdf.
It took me 5 months of multiple attempts trying to get into Lemmy.
I tried so many different instances.
.world was the only instance that would let me in.
can someone please explain to me why it's so difficult to get into some instances?
Why was world the only one I could get into?
have you tried sdf.org, the I am registered to? It's fairly easy to register last time.
Huh. I saw some list when I was joining detailing different instances and why you might join them. There was zero issue when I picked the instance I'm on (lemmy.zip), but I'm not sure if that list still exists.
Welcome to the Unpopular Opinion community!
How voting works:
Vote the opposite of the norm.
If you agree that the opinion is unpopular give it an arrow up. If it's something that's widely accepted, give it an arrow down.
Guidelines:
Tag your post, if possible (not required)
Rules:
1. NO POLITICS
Politics is everywhere. Let's make this about [general] and [lemmy] - specific topics, and keep politics out of it.
2. Be civil.
Disagreements happen, but that doesn’t provide the right to personally attack others. No racism/sexism/bigotry. Please also refrain from gatekeeping others' opinions.
3. No bots, spam or self-promotion.
Only approved bots, which follow the guidelines for bots set by the instance, are allowed.
4. Shitposts and memes are allowed but...
Only until they prove to be a problem. They can and will be removed at moderator discretion.
5. No trolling.
This shouldn't need an explanation. If your post or comment is made just to get a rise with no real value, it will be removed. You do this too often, you will get a vacation to touch grass, away from this community for 1 or more days. Repeat offenses will result in a perma-ban.
Instance-wide rules always apply. https://legal.lemmy.world/tos/