35
Anyone with experience using anubis on their lemmy server, have a good config? (github.com)
submitted 2 weeks ago by dessalines@lemmy.ml to c/lemmy@lemmy.ml
16 comments fedilink hide all child comments

I've recently added anubis to lemmy.ml, and it seems to be working well.

I have a PR to add anubis to lemmy-ansible (our main installation method), and I could use some help tweaking / optimizing its botPolicy.yaml config, for federated services.

Anyone with experience running anubis, this would be much appreciated.

you are viewing a single comment's thread
view the rest of the comments
[-] ex_06@slrpnk.net 3 points 2 weeks ago

@poVoq@slrpnk.net

[-] poVoq@slrpnk.net 6 points 2 weeks ago

This is the botPolicy.yaml that we use on slrpnk.net :

bots:
  - name: known-crawler
    action: CHALLENGE
    expression:
      # https://anubis.techaro.lol/docs/admin/configuration/expressions
      all:
        # Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
        - userAgent.contains("Macintosh; Intel Mac") && userAgent.contains("Chrome/125.0.0.0") # very old chrome?
        - missingHeader(headers, "Sec-Ch-Ua") # a valid chrome has this header
    challenge:
      difficulty: 6
      algorithm: slow

    # Assert behaviour that only genuine browsers display.
    # This ensures that Chrome or Firefox versions
  - name: realistic-browser-catchall
    expression:
      all:
        - '"User-Agent" in headers'
        - '( userAgent.contains("Firefox") ) || ( userAgent.contains("Chrome") ) || ( userAgent.contains("Safari") )'
        - '"Accept" in headers'
        - '"Sec-Fetch-Dest" in headers'
        - '"Sec-Fetch-Mode" in headers'
        - '"Sec-Fetch-Site" in headers'
        - '"Accept-Encoding" in headers'
        - '( headers["Accept-Encoding"].contains("zstd") || headers["Accept-Encoding"].contains("br") )'
        - '"Accept-Language" in headers'
    action: CHALLENGE
    challenge:
      difficulty: 2
      algorithm: fast

  - name: generic-browser
    user_agent_regex: (?i:mozilla|opera)
    action: CHALLENGE
    challenge:
      difficulty: 4
      algorithm: fast

status_codes:
  CHALLENGE: 202
  DENY: 406

dnsbl: false

#store:
#  backend: valkey
#  parameters:
#    url: redis://valkey-primary:6379/0

I think I just took it over from Codeberg.org back from when they still used Anubis. Nothing really relevant to Lemmy specifically and it is only in front of the frontends, not the s2s federation API.

It seems though like there are some crawlers that use 3rd party hosted alternative frontends to crawl (unintentionally?) through the federation API, so something in front of that would be useful I guess.

this post was submitted on 09 Apr 2026
35 points (100.0% liked)

Lemmy

14580 readers
11 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 6 years ago
MODERATORS
nutomic@lemmy.ml