Me running everything on a single postgress instance on my shitbox 0€/month
0? My energy company says I'm using power equivalent to a family of eight. And it's just wifey , the servers and me. I had cops here asking if I grow weed 😁
So unless you steal power, it surely isn't close to 0 😁
I realised I don’t need my servers being online 24/7, so for me that’s Raspberry Pi and equivalents, plus powering on computers on demand.
A trick I realized a few years ago: Caddy has a module you can build it with that does WOL. So I was able to run a Caddy reverse proxy that woke up my higher powered server on demand, and let it go back to sleep when I wasn't using it. Might be a bad idea for a database sever, but for my uses it was pretty simple and effective.
Oh wow, that’s really cool! I do use Caddy too.
Is it that your service/website is on both (low powered server and high powered one) or is it only on the high powered? So, it’s like
- the lower powered server knows it needs help (sounds a bit surreal to me, but perhaps it’s doable)
- or the lower powered server does not serve anything, but wakes up the high powered when the thing is accessed?
I guess that’s the 2nd thing, but it’s very cool indeed! That way you can really have very convenient things for free, as it’s super cheap to run any hardware for a very while on demand. I don’t mind waiting a minute or even two when I need to access something very infrequently and don’t want to run my server 24/7. I do exactly that, but I wake up it via LAN manually.
The low powered server is the Caddy server, all it does us act as a reverse proxy for everything in my house, giving it an SSL cert and doing things like WOL. The caddy config basically just says "Here's your reverse proxy target, if you don't get a response within one second, send a WOL packet, wait a couple of seconds, then try again".
The only requirement is for you to do a custom build of caddy (this is done with a dockerfile), and to have WOL enabled on the high power server.
It means the first web request for services on the high power server might take a few seconds, but everything after that is smooth.
Is it that high power server takes a few seconds to boot? What’s the hardware you have there? I’m curious that’s the average boot time for an average high power server? I do use heavily obsolete devices for my personal servers (think of DDR-2 era devices with Intel Atom or sometimes core 2 duo devices) usually without even SSDs. With an SSD, my desktop devices (all DDR-3 era with SATA-3 disks) boot within 20…30 seconds, which is good enough for me. I assume the more modern devices would be quicker, but [single-digit, I assume] seconds sounds very good. To me, that sounds like it’s a no-brainer to have this feature. I was thinking whether I can wait minutes for something I need occasionally to boot. Seconds is just too fast. I think that delay is tolerable even for a commercial / production server, where the expectations are just different.
I'm not shutting it down, just sleeping it. My high powered server is a gaming PC from a few years ago. Running Linux, my best case scenerio for cold boot would still be 10-20 seconds, but wake from sleep is near instant.
This is why im doing my homelab on low powered processors (5825u NAS boards). Runs way cooler and is way more efficient. Same performance as my 9900kf gaming PC cpu wise.
Edit: this hasn't happened yet
I was on very low-powered too. As I'm a damn cheapskate. But then the performance started to piss me off and i went nuts. Plus a fat gpu for LLAMACPP for the smarthome. Also I'm gaming A LOT. So it adds up :)
That's fair. I have a 6800xt in my pc and I use that for my llm.
That said, I think I'm slightly misleading you. My current setup is not a 5825u. It's my old laptop with an i7-8550u. I'm going to move it to the 5825u soon.
Ok so if I'm reading this correctly: They migrated from an OS and MySQL version receiving no updates since at least 2 years to MySQL 8.0 which will stop getting updates in 4 days. Also every service is running without any containerization and there is a single database for everything... and it all runs on a single host and I didn't read one word about a backup strategy or disk encryption. Also not a single word about infrastructure as code like ansible so that you can reliably recreate the system... and The whole stuff is hosted in Germany for a Turkish software company - sounds like very good latency.
My personal conclusion: This system WILL fail and the guy who designed it is stuck somewhere 10-20 years in the past.
Sounds like my homelab has better redundancy than these guys, and my monthly bill isn't much different than their new one. I only pay for power and networking, since I own my own hardware. I'm colocating in my city, so my latency to home is about 1ms, and I've got a full mirrored server in my house. Certain files are further backed up elsewhere for proper 3-2-1 backup (+ each server running raidz2 with disk encryption). Even if my home Internet goes out, I still have full access to my files at home, and all my public services stay running in the data center. If either server fails, it's all set up with containers so it's easy to spin up each service somewhere else.
One thing that's tricky to get right with disk encryption (especially with encrypted /boot) is having a redundant boot partition. I was able to hack this together by having sofware raid duplicate my boot partition to a second drive. Now if I remove either OS boot drive it falls back to the remaining one. To prevent breaking EFI boot, you need to use the Version 1 RAID format so the metadata is stored at the end of the partition, not the front where EFI reads.
I'm in the US and when I tried migrating from DO to Hetzner, I got asked to upload my passport to prove I'm not spam or something. Same experience with OVH.
Is this a thing for all European hosting companies? I ended up finding some Canadian hosting that would just let me sign up and pay like normal.
Lots of respectable EU hosting companies, and also aparently OVH, if they think there's a chance you're taking the piss will ask for a ID so they can ban you. It's not just anti-spam, it's anti-abuse and for preventing non-payment. They think there was a risk involved in accepting your business (whatever that may be, obviously companies don't dilvulge their criteria here), and if you go elsewhere they're not upset about it for that reason.
I never had that kind of experience with Hetzner or OVH as a European. I suppose there are extra hoops to jump through for US customers for some reason?
Is this a thing for all European hosting companies?
Absolutely not. At least not in Europe.
Have you tried netcup as well?
Netcup was the one I had most problems years ago about uploading identify check. Last year when I signed again they actually put a system in place that it's simpler, you just show your face for some photos and show your identity card and it checks if it matches. So an external identity provider. Simpler than having to see how to upload copies by email with pgp (which they support and have documentation about).
It's so weird. Where from? I never had any such requirements with any provider, even when I, from Europe, bought something from abroad.
I'm in europe. If you put a name that may seem a bit non standard they will fire that verification immediately. I think i tried hetzner before and same thing. On ovh was no problem for the little time I tried. Anyway they all always want full physical address and name and a bunch of personal details.
Europe or EU?
If you put a name that may seem a bit non standard they will fire that verification immediately.
You mean you also tried with a standard name? What exactly is "non standard"? How exactly is it "fired"?
My name isn't standard either.
Is there anything else out of the ordinary in your application? Like, payment method, no/invalid smartphone, no/invalid email, no/invalid mailing address, etc.?
I repeat, I had to do no identity check whatsoever. Neither with one other European provider I used.
Really good blogpost, as a sysadmin, this is a great way to handle a migration with zero downtime.
When I was migrating my servers to NixOS I did the same thing, I tried to make my configuration the same as the old OS so everything works cleanly, and it worked fine, but since it was all in the same server, I had to do manual migration for things like files and databases.
Once the dump was complete, we transferred it to the new server using rsync over SSH. With 248 GB of compressed chunks, this was significantly faster than any other transfer method:
rsync -avz --progress /root/mydumper_backup/ root@NEW_SERVER:/root/mydumper_backup/
that's a bit weird. rsync -z is compression, but they did compress in the mydumper export already, so this is a slow down (or neutral at best). also in my experience rsync is as fast as scp is as fast as piping anything to the tcp port on the destination etc. rsync does not win for speed but for enabling resume so to say...
besides this: nice read!
Not a sysadmin but just an hobbyist: is it ok to have such a large install bare metal and not containerized?
For example the issue of MySQL 5 being unavailable would be a non-issue with a container
Totally fine. Containerization comes at a cost too. It's a matter of system design, knowing your risks and complexities, and handling them accordingly.
With such a size, before containerization I'm wondering if these services are not independent enough to split them onto multiple servers.
Having everything together reduce system complexities in some ways, but not in other ways.
Wha?
You do realize there are plenty of bare metal infrastructure deployments out in the world, yeah? Being in a container solves no problems in this scenario at all.
you can just set up containers on your bare metal server. in fact if you're going to install insecure services you definitely want to containerize them, though tbh you need to run really far away from whatever it is you're doing that requires sql5, or at least don't let it be reachable on the internet, that should be network-isolated, which really limits its utility.
in fact if you're going to install insecure services you definitely want to containerize them,
While this is true, if you're running a platform that is root by default (looking at you, docker), you're not shielding yourself as much as you might think you are.
If you're running an insecure app as root, you better hope they don't also have an exploit to get out of the container after the app is popped, otherwise you're fucked.
Technology
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.