209
Is Lemmy's code base "good"?
(lemm.ee)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 03 Jul 2023
209 points (97.7% liked)
No Stupid Questions
35843 readers
308 users here now
No such thing. Ask away!
!nostupidquestions is a community dedicated to being helpful and answering each others' questions on various topics.
The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:
Rules (interactive)
Rule 1- All posts must be legitimate questions. All post titles must include a question.
All posts must be legitimate questions, and all post titles must include a question. Questions that are joke or trolling questions, memes, song lyrics as title, etc. are not allowed here. See Rule 6 for all exceptions.
Rule 2- Your question subject cannot be illegal or NSFW material.
Your question subject cannot be illegal or NSFW material. You will be warned first, banned second.
Rule 3- Do not seek mental, medical and professional help here.
Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.
Rule 4- No self promotion or upvote-farming of any kind.
That's it.
Rule 5- No baiting or sealioning or promoting an agenda.
Questions which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.
Rule 6- Regarding META posts and joke questions.
Provided it is about the community itself, you may post non-question posts using the [META] tag on your post title.
On fridays, you are allowed to post meme and troll questions, on the condition that it's in text format only, and conforms with our other rules. These posts MUST include the [NSQ Friday] tag in their title.
If you post a serious question on friday and are looking only for legitimate answers, then please include the [Serious] tag on your post. Irrelevant replies will then be removed by moderators.
Rule 7- You can't intentionally annoy, mock, or harass other members.
If you intentionally annoy, mock, harass, or discriminate against any individual member, you will be removed.
Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.
Rule 8- All comments should try to stay relevant to their parent content.
Rule 9- Reposts from other platforms are not allowed.
Let everyone have their own content.
Rule 10- Majority of bots aren't allowed to participate here.
Credits
Our breathtaking icon was bestowed upon us by @Cevilia!
The greatest banner of all time: by @TheOneWithTheHair!
founded 1 year ago
MODERATORS
It’s decent, but it isn’t scalable, at least not yet.
Right now the entire Lemmy backend is one big “monolith”. One app does everything from logins and signups to posting and commenting. This makes it a little harder to scale, and I wouldn’t be surprised to see it split out into multiple micro services sooner rather than later so some of the bigger instances can scale better.
I’d love to know where the higher level dev stuff is being discussed and if they’ve made a decision on why or why not microservices.
Microservices aren't a silver bullet. There's likely quite a lot that can be done until we need to split some parts out, and once that happens I expect that federation would be the thing to split out as that's one of the more "active" parts of the app compared to logins and whatnot.
Definitely not a silver bullet, but should stop the app from locking up when one thing gets overloaded. I’m sure they have their reasons for how it’s designed now and I’m probably missing something that would explain it all.
I’m still not familiar enough with how federation works to speak to how easy that would be. Unfortunately this has happened all as I’ve started moving and I haven’t gotten a chance to dive into code like id want to.
It's also not the only solution for high-availability system. Multiple monoliths with load-balancing can be used as well.
Also, a lot of people are self-hosting. In this case, microservice won't give them any scaling benefit.
The problem with scaling monoliths is you are scaling everything, including the pieces that have lower usage. The huge benefit you get from going to micoservices is you only have to scale the pieces that need to be scaled. This allow for horizontal scaling to use less compute resources. It also allow for these compute resources to be spread out as well.
A lot of the headaches can be removed by having an effective CI/CD strategy that is completely reusable with minimal effort.
The last headache would be observability. There you're stuck either living with the nightmare of firefighting problems with 100 services in possibly 10 locations, rolling your own platform using FOSS tools or spending a whole lot of money on something like honeycomb, datadog or new relic.
But I'm an SRE, I live my life for scalablability and DevOps processes. I know I'm biased.
I find that often the overhead from microservices is worse than any savings from dropping a megabyte worth of unused machine instructions from a binary.
When your microservices need to talk to each other (and I'm not sure how many services you could split out of Lemmy without them needing to talk to each other), you're doing a bunch of HTTP requests that are way slower than just calling another function in your monolith.
I see this at work every day. We run a distributed monolith because someone thought microservices would be a good idea, but we can't actually separate everything, so it's usual for an incoming API call to make 2-3 more calls internally. It can get so, so slow.
Overly chatty micoservices are definitely an issue.
Changing your mindset to a microservice oriented architecture is not an easy feat, it's something that took a lot of time for me to fully grasp (back in my architect/developer) days. Yes, you gain overhead that will need to be compensated for. But when do the benefits outweigh the disadvantages?
Here are some questions to ask during design: How much of this chattiness is because you are tightly coupling these services? Hom much should a microservice be talking between each other? Can you implement an event bus to handle that chatter between services?
Designing an application using microserves but just replicating the monolith application will give you scalability, but will not give resilience. What can you do to overcome that single point of failure? First, no more synchronous calls to these APIs, toss an event over then fence and move on. Degrade your application if the failure is something you can't overcome, but don't just stop the application because one API is no longer responsive.
Do you need everything to be a microservice? Probably not. The first thing you look at when moving from a monolith to microservice architecture is what makes the most sense to be moved. How much work can be offloaded to background jobs (using something like sidekiq)?
How do you handle installs? How many packages do we now have to create for this application to work?
There are a lot of questions that have to be answered before moving toward a microservice architecture. On top of that, there is a complete mindset change as to how the application works that needs to be accomplished. If you design your microservice application with a monolith application mindset, you'll never realize any of the gains by making the move
I'm not into this topic, but i do want to share a talk by Joe Armstrong (creator of erlang).
https://www.youtube.com/watch?v=cNICGEwmXLU
The whole chanel is full of stuff like this.
Scaling monoliths still works fine though. Microservices are first and foremost an answer to an organizational problem, not a technical one. There is a very high chance that if you are doing microservices with less than 20 people, or let’s say even 50 people, you are doing it wrong.
Microservices introduce a ton of overhead in engineering effort required, which needs to be balanced with the benefit they provide.
Scaling shouldn’t be the first and only reason for doing microservices.
The admin of lemm.ee is currently scaling things horizontally - it is possible.
Do you have more details about this? Lemmy and lemmy-ui are stateless and can be easily scaled horizontally, but pictrs is not stateless and use a filesystem-based database (sled) with lock and it can only be run as a single replica or it crash (if the replica run in the same host and can't acquire the lock) or will have severe data inconsistency issue (if the replica run in separate host with separate sled database file).
I'm a software engineer myself, but not familiar with your field. How would your practice be applied to self-hosting? I'm assuming a bunch of people with their home servers wouldn't want to just run OpenShift.
Personally, I wouldn't touch OpenShift. As someone that has a kubernetes cluster hosted at my house on a mixture of RPis, a nas and in VMs, I'm not one to to say what anyone else would do :).
But, that can be overcome, it's all about designing you application for multiple different installs. You don't have to have all your services running fully separately. You can containerize each service and deploy to an orchastration engine such as kubernetes or docker swarm, or you can have the multiple endpoints on a single machine with an install package that keeps them together. It's all about architecting toward resiliency, not toward a pattern for no other reason.
Also, Google has some very good books on SRE for free.
I've been wanting to check out Kubernetes! Thank you!
Microk8s or k3s are the implementations I'd start with. Easy install and gets you started with something without a lot of the work.