Website operators are being asked to feed LLM crawlers poisoned data by a project called Poison Fountain.
The project page links to URLs which provide a practically endless stream of poisoned training data. They have determined that this approach is very effective at ultimately sabotaging the quality and accuracy of AI which has been trained on it.
Small quantities of poisoned training data can significantly damage a language model.
The page also gives suggestions on how to put the provided resources to use.
Seems like a bad take from my POV, as someone who uses and has made money using LLMs I feel is not ok to poison them, I wouldn't feel ok with myself getting something for free and even gain money with and at the same time be poisoning it so my take will be: you can always block crawlers in your nginx.conf with some extra steps, you can even use an LLM to do it for you and improve to block all major crawlers. IMHO if it's public data is even public for crawlers is up to you if you set up a block for these on your behalf.
"Public" is a tricky term. At this point everything is being treated as public by LLM developers. Maybe not you specifically, but a lot of people aren't happy with how their data is being used to train AI.
Also they always come up with new ways to circumvent blocking mechanisms and push some extra work to admins.
Remember how judges ruled when somebody circumvented copy restrictions on media?