this post was submitted on 08 Aug 2025
413 points (99.5% liked)
Privacy
40679 readers
439 users here now
A place to discuss privacy and freedom in the digital world.
Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.
In this community everyone is welcome to post links and discuss topics related to privacy.
Some Rules
- Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
- Don't promote proprietary software
- Try to keep things on topic
- If you have a question, please try searching for previous discussions, maybe it has already been answered
- Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
- Be nice :)
Related communities
much thanks to @gary_host_laptop for the logo design :)
founded 5 years ago
MODERATORS
My instance gets pillaged once a day for 20 minutes by what I think is a scraper for an LLM.
The scraper grabs every post and profile page and the load on the server triggers alerts but the site stays usable.
I haven't been able to put a stop to it as the requests come from 1500+ IP addresses, with different user agents.
Run your access logs through something that will report the ASN for the client IPs. Goaccess would be my recommendation. It will require access to a GeoIP database which you can get from Maxmind by signing up for a free API key, or download them directly from P3TERX/GeoLite.mmdb on Github. We have identified a number of bot networks this way. Happy to help further if you'd like a hand ๐
Yeah, they're scraping alright and it's all purposefully done in such a way that you can't stop it, you can't control it.
AI companies are criminal as far as I am concerned
Anubis?
I have no idea. I spot check 20 or so IP addresses and they are all from different AS networks. Truly diverse botnet. Feel powerless.
they were suggesting a solution, this proof-of-work web firewall: https://github.com/TecharoHQ/anubis
Ah thank you, will check it out
I think Anubis would be able to prevent that. Sopuli uses it
Thanks I'll have a look