We need mod_Anubis directly added to Nginx or Apache with configuration options like allowing or blocking specific URLs, IPs, CIDRs, or even data center ranges like AWS. additionally, customization for traps and stuff like that. Freeloader AI companies abuse open-source projects, small businesses, blogs, forums, and artists without giving back to communities or individuals. They are making billions while people are left with server bandwidth bills.
@nixCraft@mastodon.social There exists https://github.com/simon987/ngx_http_js_challenge_module, a mod for Nginx that works similar to Anubis.
@nixCraft@mastodon.social I would love to see an haproxy module
@nixCraft@mastodon.social I'd love this. My apache/fail2ban config is just chasing its tail since the IP addresses keep moving. I block every IP I find and get new ones the next day.
@nixCraft@mastodon.social the JWT should be validated in the reverse proxy. Anubis would be a perfect authorization endpoint to point to when required. But I think the logic should be present in more server side frameworks. PHP for the masses, container image for classic anubis and embeddable libraries for rust or java services…
@nixCraft@mastodon.social I wonder if @CrowdSec@infosec.exchange would be helpful in this case.
@nixCraft@mastodon.social Do you know if there is currently support for github pages?
@nixCraft@mastodon.social well actually it's mainly just proof of work in between you the client and the server. So bots do not like proof of work.
@nixCraft@mastodon.social
@nixCraft@mastodon.social
Meta's AI has already overcome this.
The way Meta is doing it is by running a bunch of mini instances, running real web browsers (just like you), and their AI scraps from within the browser. Each mini instance also has a unique IP and some random browser history to help it pass as human. The AI can also simulate random mouse movements and bypass (solve) Captcha.
However, the good news, at least for now, is it is slower than the traditional scraping method. And thankfully, most of the other AIs out there do not yet go to such extremes. It is costing Meta a fortunate to run their little pilot program.
@nixCraft@mastodon.social @esvrld@normal.style Here.
@nixCraft@mastodon.social Anubis is a great step forward in defense against bad bots. Nepenthes takes it a fun step further and adds some teeth. :)
@nixCraft@mastodon.social I added Anubis to my website hofstede.io yesterday. Was simple and works like a charm. Just deployed it as a container and included it into my traefik configuration 🙂
According to my logs, the crawler traffic did decline significantly! 😊
An umbrella community for all things cybersecurity / infosec. News, research, questions, are all welcome!