Anyone know what's happening with lemmy.ca? (sh.itjust.works)

submitted 1 week ago by JohnnyCanuck@sh.itjust.works to c/fediverse@lemmy.world

20 comments fedilink hide all child comments

top 20 comments

sorted by: hot top controversial new old

[-] Shadow@sh.itjust.works 65 points 1 week ago* (last edited 1 week ago)

Our hardware failed this morning and we're stuck waiting on OVH to investigate the support ticket I filed. They claim their support is 24/7 but live chat / phone is just business hours, so worst case we might be down until tomorrow =(

https://status.lemmy.ca/ has an update, but not much to say really except "It's down until it's not"

Edit: I'm assuming we're down until I can call / chat with them first thing in the morning, so expect to be back up early tomorrow (depending on how long the parts replacement takes).

Edit 2: I called them when I woke up at 6:30 PST and hopefully they'll fix it soon now....

[-] Bakkoda@sh.itjust.works 31 points 1 week ago

Just changed all my maintenance messages to

It's down until it's not

[-] Brewchin@lemmy.world 15 points 1 week ago

They claim their support is 24/7 but live chat / phone is just business hours

It's increasingly common for "24x7 support" to mean access to their KB/FAQ and the ability to raise a ticket, while the actual support is a skeleton crew working regular hours.

Not unlike "your call is very important to us..." 😅

[-] AnUnusualRelic@lemmy.world 8 points 1 week ago

You can send us email at any time of day or night!

[-] JohnnyCanuck@sh.itjust.works 7 points 1 week ago

Thanks! I'll bookmark that.

[-] ininewcrow@lemmy.world 15 points 1 week ago

Weird to think what happens when you lose a site you visit on a regular basis. I can understand the whole instance going down but us users just get lost when things get turned off. There's no way to know what happened, why or when or if anything will return.

I had to resurrect my old lemmy.world account to figure out what happened. I guess we have to hang on to two or more unique instance accounts in case our main instance goes down.

Stay well guys and I totally appreciate the admins of lemmy.ca and all the work they do. I'll patiently wait and look forward to seeing it online again.

[-] Shadow@sh.itjust.works 9 points 1 week ago

I keep trying to open the lemmy app on my phone out of habit.

[-] ininewcrow@lemmy.world 6 points 1 week ago

It feels shitty and I was doing the same thing all day. I had to resurrect my lemmy.world account just to get back online. It's made me realize that I should maintain both accounts for shut downs like this. And now I'm also having to do more in maintaining my Mastodon account too.

[-] JohnnyCanuck@sh.itjust.works 3 points 1 week ago

I did that five times before remembering I had an old account on this instance that I could switch to to post the question and see what communities I used to be subscribed to.

[-] Aqua@lemmy.vg 7 points 1 week ago

At least it’s not like Reddit when the website goes down you can’t use it at all.

[-] poVoq@slrpnk.net 10 points 1 week ago

They posted on the admin chat that their hosting provider had a hardware failure and is working on providing a replacement server.

[-] Seigest@lemmy.world 9 points 1 week ago

It may not get fixed untill tommrow since they are not getting any response from their service provider.

[-] hendrik@palaver.p3x.de 7 points 1 week ago* (last edited 1 week ago)

That's an error message from Cloudflare, their tunnel provider. The server is probably unreachable. Either they're doing some maintenance, or it's a hiccup or something went wrong. You need to wait for a moment. Or maybe a bit longer if they're facing any issues.

[-] hendrik@palaver.p3x.de 3 points 1 week ago* (last edited 1 week ago)

Wow, they're really offline for a long time now. Did OVH really mess up that hard? I guess hardware in a datacenter fails every day? Wouldn't they be prepared for that and just restore/migrate to a different machine, then?

[-] catloaf@lemm.ee 2 points 1 week ago

Yeah that's usually how it's supposed to work, but sometimes it doesn't work properly for a variety of reasons.

[-] hendrik@palaver.p3x.de 3 points 1 week ago* (last edited 1 week ago)

Yeah, it's speculation at this point anyways. I mean I've only ever seen webservers with a RAID. So one harddrive failure shouldn't affect it too much. Then most people rent a VPS and that usually runs on some cluster. Even if a whole machine goes down, it's supposed to come up again on some other machine automatically within a few minutes. And then you should have tech support of a big hosting provider answer within some timespan. I hope someone will write a summary of the events and link it somewhere. Maybe a lot of things went wrong. Or it's some kind of error in the specific setup lemmy.ca did.

[-] Shadow@sh.itjust.works 3 points 1 week ago

It wasn't a hard drive failure but something more severe, since the remote console is non-responsive. I just called OVH to yell at them and press on the ticket, so hopefully should be resolved soon.