Whats confusing the hell out of me is: why are they bothering to scrape the git blame page? Just download the entire git repo and feed that into your LLM!

9/10 the best solution is to block nonresidential IPs. Residential proxies exist but they're far more expensive than cloud proxies and providers will ask questions. Residential proxies are sketch AF and basically guarded like munitions. Some rookie LLM maker isn't going to figure that out.

Anubis also sounds trivial to beat. If its just crunching numbers and not attempting to fingerprint the browser then its just a case of feeding the page into playwright and moving on.

Wow that was a frustrating read. I dd not know it was quite that bad. Just to highlight one quote

they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. [...] If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

This is the most crazy read on subject in a while. Most articles just talk about hypothetical issues of tomorrow, while this one actually full of today's problems and even costs of those issues in numbers and hours of pointless extra work. Had no idea it's already this bad.

How much you wanna bet that at least part of this traffic is Microsoft just using other companies infrastructure to mask the fact that it’s them

Sometimes, I hate humanity.

I'm perfectly fine with Anubis but I think we need a better algorithm for PoW