41
The Backlash Against AI Scraping Is Real and Measurable
(www.404media.co)
While many companies respect robots.txt instructions, some do not. Several large companies, including Perplexity, have been caught using proxies and surreptitious user agents to circumvent and ignore robots.txt.
respecting robots.txt already died
robots.txt has been dead for over a decade. Massive web scraping is big business at this point.
For sure, I just mean in the context of AI since OP suggested it wouldn't be respected much longer
On the road to fully automated luxury gay space communism.
Spreading Linux propaganda since 2020
Rules: