52
submitted 8 months ago* (last edited 8 months ago) by Dark_Arc@lemmy.world to c/selfhosted@lemmy.world

My main account is dark_arc@social.packetloss.gg. However, as of roughly ~~24-hours ago~~ (it seems this has been going on since March 10th and gotten worse since) it seems like the server has stopped properly retrieving content from lemmy.world.

It's been running smoothly for well over 9 months, and (I think) working fine for content coming in from other instances. So I'm curious if anyone else experienced anything strange with lemmy.world federation recently?

Setup Description

The server flow in my case is as follows:

[Public Internet] <-> [Digital Ocean Droplet] <-> [ZeroTier] <-> [Physical Machine in my Basement (HW Info)]

The Digital Ocean droplet is a virtual host machine that forwards requests via nginx to the physical machine where a second nginx server (running the standard lemmy nginx config) then forwards the request to the lemmy server software itself.

Current Status

Lemmy Internal Error

I've found this is my lemmy logs:

2024-03-24T00:42:10.062274Z  WARN lemmy_utils: error in spawn: Unknown: Request limit was reached during fetch
   0: lemmy_apub::objects::community::from_json
             at crates/apub/src/objects/community.rs:126
   1: lemmy_apub::fetcher::user_or_community::from_json
             at crates/apub/src/fetcher/user_or_community.rs:87
   2: lemmy_server::root_span_builder::HTTP request
           with http.method=POST http.scheme="http" http.host=social.packetloss.gg http.target=/inbox otel.kind="server" request_id=688ad030-f892-4925-9ce9-fc4f3070a967
             at src/root_span_builder.rs:16

I'm thinking this could be the cause ... though I'm not sure how to raise the limit (it seems to be hard coded). I opened an issue with the Lemmy devs but I've since closed it while gathering more information/making sure this is truly an issue with the Lemmy server software.

Nginx 408 and 499s

I'm seeing the digital ocean nginx server reporting 499 on various "/inbox" route requests and I'm seeing the nginx running on the physical machine that talks directly to lemmy reporting 408 on various "/inbox" route requests.

There are some examples in this comment: https://lemmy.world/comment/8728858

you are viewing a single comment's thread
view the rest of the comments
[-] Demigodrick@lemmy.zip 8 points 8 months ago* (last edited 8 months ago)

Here's a handy site to check your federation status: https://phiresky.github.io/lemmy-federation-state/site?domain=social.packetloss.gg

It looks like all the sites are lagging, which can sometimes mean there is an issue. What's the specs of the server? Can you try restarting it and seeing if that helps?

There are issues for servers hosted in places like Australia because of the latency in communicating with .world due to how big it is and how much data it sends because of this.

[-] db0@lemmy.dbzer0.com 2 points 8 months ago

This is the right answer

[-] SeeJayEmm@lemmy.procrastinati.org 1 points 8 months ago

I'm not having issues that I'm aware of, but that site always returns Network Request Failed and I haven't figured out why.

[-] Dark_Arc@lemmy.world 1 points 8 months ago* (last edited 8 months ago)

So, I think this is a (helpful) general comment but wrong in this/my specific case.

The server is so small it's not really going to register on a 10-minute frequency for outgoing content -- I'm not that much of a lemmy addict! haha.

You can see in a comment here my most recent comment to lemmy.world did sync: https://lemmy.world/comment/8728858

I'm not having any issues with outgoing content, beehaw, the KDE instance, and several others. It's just lemmy.world that's acting up (which is unfortunately because it's my favorite -- I mod/run several communities and donate to here/them -- haha).

this post was submitted on 23 Mar 2024
52 points (94.8% liked)

Selfhosted

40394 readers
407 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS