overview for cole

Basically, the lemmy backend service for some reason marked every instance we federated with as inactive, which caused it to stop outbound federation with basically everyone. I have a few working theories on why, but not fully sure yet.

TL;DR lemmy bug, required manual database intervention to fix

This was a stressful start to a vacation!

For a more detailed working theory...

I've been doing a lot of infrastructure upgrades lately. Lemdro.id runs on a ton of containerized services that scale horizontally for each part of the stack globally and according to load. It's pretty cool. But my theory is that since the backend schedules inactive checking for 24 hours from when it starts that it simply was being restarted (upgraded) before it had a chance to check activity until it was too late.

theory:

scheduled task checks instances every 24 hours
I updated (restarted it) more than every 24 hours
it never had a chance to run the check
???
failure

This isn't really a great design for inactivity checking and I'll be submitting a pull request to fix it.

1

Recent Downtime (lemdro.id)

submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/lemdroid@lemdro.id

0 comments fedilink

There was a brief (~5 minute) period of downtime on lemdro.id recently. This occurred during a routine database upgrade when some strange issue caused a member of the cluster to become inconsistent and refuse to form quorum.

This locked the entire cluster into an invalid state which took some troubleshooting to fix. My apologies.

I will be rolling out read replicas for folks on the East coast of the US as well as those in Europe sometime in the next week, you should notice a pretty dramatic reduction in latency if you are from those areas. Additionally, other recent changes have increased reliability and decreased latency which may or may not be noticeable.

I will post another update before I start rolling out the read replicas since it is kind of a big change (and I will schedule a time for it)

1

Infrastructure Upgrade Rollback (lemdro.id)

submitted 2 years ago by cole@lemdro.id to c/lemdroid@lemdro.id

0 comments fedilink

I recently rolled out an infrastructure upgrade this last Monday (Dec 4) with the intent of reducing peak response times and removing occasional scaling errors.

Unfortunately, my metrics system showed slightly elevated error rates, so I've decided to rollback these changes for now. I will make another announcement before I roll them back out in the future. Thanks for your support!

1

Google Pixel Update - November 2023 - Google Pixel Community (support.google.com)

submitted 2 years ago by cole@lemdro.id to c/googlepixel@lemdro.id

0 comments fedilink

Phishing Mails by cole in c/lemmyworld@lemmy.world

[-] cole@lemdro.id 18 points 2 years ago

I've gotten an email like this before for lemdro.id. I think it's a generic phishing email since the community links look like email addresses (and actually often are)

1

Should lemdro.id change its default instance to Photon? Please vote! (lemdro.id)

submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/lemdroid@lemdro.id

0 comments fedilink

Hey there everyone. I think the Photon project has matured enough to the point where I feel ready to replace the default lemmy frontend with it. Since this instance serves roughly 1000 people now, I figured this was worth holding a vote on!

Please check out Photon as currently hosted at https://p.lemdro.id.

If you support changing the default frontend to Photon, upvote my comment on this post. If you don't support it, downvote that same comment.

Thanks!

Windows 11 vs Linux supported HW by cole in c/linux@lemmy.ml

[-] cole@lemdro.id 19 points 2 years ago

Freesync yes, HDR soon™

8

Chromecast Audio appreciation post (lemdro.id)

submitted 2 years ago* (last edited 2 years ago) by cole@lemdro.id to c/androidmemes@lemdro.id

9 comments fedilink

RIP :(

4

The Company Behind Stable Diffusion Appears to Be Crumbling Into Chaos (futurism.com)

submitted 2 years ago by cole@lemdro.id to c/aistuff@lemdro.id

0 comments fedilink

291

Android rule (lemmy.blahaj.zone)

submitted 2 years ago by cole@lemdro.id to c/androidmemes@lemdro.id

17 comments fedilink

[META] The merger from lemmy.world has locked out k.bin members. [FIXED] by cole in c/android@lemdro.id

[-] cole@lemdro.id 20 points 2 years ago

sure! take a look at this issue: https://github.com/LemmyNet/lemmy-ansible/issues/106

It is likely yours is fixed if you deployed recently

[META] The merger from lemmy.world has locked out k.bin members. [FIXED] by cole in c/android@lemdro.id

[-] cole@lemdro.id 88 points 2 years ago

Hello! Admin here at lemdro.id. This is the result of several problems in the default reference nginx config. I am working on resolving this right now and should have it fixed within the next 30 minutes!