1589
Black Mirror AI
(mander.xyz)
A place for majestic STEMLORD peacocking, as well as memes about the realities of working in a lab.
Rules
This is a science community. We use the Dawkins definition of meme.
Web manager here. Don't do this unless you wanna accidentally send google crawlers into the same fate and have your site delisted.
Wouldn't Google's crawlers respect robots.txt though? Is it naive to assume that anything would?
It's naive to assume that google crawlers respect robot.txt.
It'd be more naive to have a robot.txt file on your webserver and be surprised when webcrawlers don't stay away. ๐