127

Cloudflare offers to make AI pay to crawl websites (www.computerworld.com)

submitted 3 days ago by Zerush@lemmy.ml to c/privacy@lemmy.ml

29 comments fedilink hide all child comments

top 29 comments

sorted by: hot top controversial new old

[-] AceFuzzLord@lemmy.zip 7 points 2 days ago* (last edited 2 days ago)

This idea that you could have Cloudflare help by telling off AI crawlers sounds nice, but how long until it becomes a premium feature that requires loads of money to operate because AI companies lobby them to make it inaccessible to the masses? Or something equally as bad happens?

[-] Ulrich@feddit.org 10 points 3 days ago

I don't believe they have that capability

[-] Zerush@lemmy.ml 31 points 3 days ago* (last edited 3 days ago)

Oh yes, they have and more, they are one of the most powerfull security and AI company with a ton of services. Perfectly capable to remove the plug of any service and web. Sadly with similar privacy concerns as Google.

https://www.cloudflare.com/

[+] Ulrich@feddit.org -14 points 3 days ago

Okay great. Go ahead and explain to me how they plan to fight an army of bots doing everything they can to be invisible?

[-] HelloRoot@lemy.lol 35 points 3 days ago* (last edited 3 days ago)

they can already block VPN traffic (unless you use their VPN)
their whole business model is based on them being a man in the middle that decrypts ssl and analyses the packets plainly
about a third of the worldwide websites are using cloudflare so they have a pretry good birds eye view on behaviour of any machine, datacenter or ip range that will be visiting a lot of websites, which in turn will trivially whether it is normal user behaviour or a crawler.

[-] Zerush@lemmy.ml 10 points 3 days ago* (last edited 3 days ago)

Not the first time that, with all my privacy measures on, instead of an page, I see the page from Cloudflare analyzing if I am a bot, before it let me access the page I wanted. Invisible in the web is only a bad joke. Anybody is visible in the moment he goes online, irrelevant if he use VPN, TOR or whatever, this times have passed. Believing it is as hilarious as in the Movie Independence day infecting with an Virus an Alien Mothership, using an crappy Laptop (I have laughed a lot with this scene).

[-] slackness@lemmy.ml 7 points 2 days ago

Anybody is visible in the moment he goes online, irrelevant if he uses ..., TOR

[-] Zerush@lemmy.ml 2 points 2 days ago

Yes, TOR never was secure against secret services and goverments, les nowadays with AI and massive server power from these. Don't forget who developed the TOR network and from whom are the servers used. Drug Barons since time turned to use pen and paper for their orders and communication, because the web and even the Dark Web isn't really private anymore (traffic analysis, exploiting software vulnerabilities, monitoring exit nodes, using Honeypot nodes......)

[-] slackness@lemmy.ml 1 points 2 days ago

Just calling it "Dark Web" gives away you have no idea what you're talking about.

[-] Zerush@lemmy.ml 2 points 2 days ago

Naturally the Darkweb has nothing to do with the TOR network, But even if you access it using the TOR network, they find you if they want, as you can see.

[+] Ulrich@feddit.org -8 points 3 days ago

they can already block VPN traffic unless it goes through their VPN

Yeah that's how most VPNs work.

their whole business model is based on them being a man in the middle that decrypts ssl and analyses the requests plainly

Okay? Analyze all you want. They can't stop bots on any of the other sites they regulate either.

about a third of the worldwide websites are using cloudflare so they have a pretry good birds eye view on behaviour of any machine that will be visiting a lot of websites

Great. Bots intentionally change up their behavior and identifying information as to be undetected.

[-] PowerCrazy@lemmy.ml 8 points 3 days ago* (last edited 3 days ago)

They can’t stop bots on any of the other sites they regulate either.

Why not? They are doing edge caching, they can literally just block the connection from visiting the site just like they do with their DDoS mitigation.

[-] Ulrich@feddit.org -2 points 3 days ago

they can literally just block the connection

block which connection? Again, these AI companies know people don't want them crawling their sites and they do everything they can to be invisible. This has been an issue for years at this point.

just like they do with their DDoS mitigation

blocking DDoS is trivial by comparison.

[-] HelloRoot@lemy.lol 8 points 3 days ago

They can’t stop bots on any of the other sites they regulate either.

They can and do. What is blocked depends on what the website owner sets as settings in cloudflare.

Bots intentionally change up their behavior and identifying information as to be undetected.

If they have to crawl the web while behaving like a normal human, it will be magnitudes slower and more costly.

[-] Ulrich@feddit.org -2 points 3 days ago

What is blocked depends on what the website owner sets as settings in cloudflare.

And how does the owner know which connections are bots?

If they have to crawl the web while behaving like a normal human, it will be magnitudes slower and more costly.

They don't care, they have trillions of dollars of VC money to power through.

[-] HelloRoot@lemy.lol 3 points 3 days ago

The owner sets the level. If they set strict level, all bots are blocked.

They do care. VC funding happens because the result is profitable. If it is less profitable, there will be less funding because of higher investment risk.

[-] Ulrich@feddit.org 0 points 2 days ago

If they set strict level, all bots are blocked.

I don't know what you don't understand. These bots are not labeling themselves as bots. They are camouflaging themselves to look like any other type of traffic.

VC funding happens because the result is profitable.

No, VC funding happens because investors are duped into thinking the result is profitable.

[-] HelloRoot@lemy.lol -2 points 2 days ago* (last edited 2 days ago)

You keep pulling bullshit statements out of your ass.

https://www.nber.org/digest/may01/how-high-are-vc-returns

VC funding IS profitable in general.

https://www.theaustralian.com.au/business/technology/titanium-ventures-builds-ai-tools-to-unearth-next-1bn-tech-unicorn/news-story/5e36720b2977704ce672ea4c3bfc1321

AI VC funding IS also profitable overall.

[-] Ulrich@feddit.org 2 points 2 days ago* (last edited 2 days ago)

I'm not and your tone is completely unnecessary so maybe dial it back a bit.

Yes, VC funding can be profitable. It's also often not. Like any other investment. Corporations will absolutely lie and blow smoke up their ass if they think it can get them more money.

[-] Steve@communick.news 9 points 3 days ago* (last edited 2 days ago)

It's literally what their entire business is based on. Filtering good and bad traffic.

[-] Ulrich@feddit.org -3 points 3 days ago

Their business is largely based on security and preventing specific types of actions on a site. Not just the mere act of visiting it.

[-] surjomukhi@lemmygrad.ml 7 points 2 days ago

It is. And they are blocking AI crawler for a while now. https://www.wired.com/story/cloudflare-tools-detect-block-ai-bots/

[-] sudo@programming.dev 0 points 1 day ago

How about you just read up on Cloudflare Turnstile instead of acting like you know anything? Here are some notable methods:

Residential IP requirements
TLS Fingerprinting
Canvas Fingerprinting

It's still possible to get around these but it's not easy. You either must have as good network engineers on staff as Cloudflare or pay some third party service to unlock it for you. All Cloudflare needs to do is keep their prices lower than the third party services.

[-] webghost0101@sopuli.xyz 0 points 2 days ago* (last edited 2 days ago)

Based on the headline this is not about blocking ai scrappers but by making them pay to do it.

Based on the discussion below which moved that goalpost the most likely answer is by making it cheaper to scrape “legally” then it costs to mimic millions of individual residential browsers with human users.

I don't know how many aces cloudflare has up its sleep to detect secret ai but they definitely have the tools to make it pretty costly and difficult. There is also a broadband impact difference between a few capitalist megapigs scrapping secretly versus loads of global basement dwellers and smaller companies scrapping worry free.

[-] SheeEttin@lemmy.zip 1 points 2 days ago

Believe all you want, reality doesn't care.

[-] Ulrich@feddit.org -1 points 2 days ago

I agree!

[-] Hirom@beehaw.org 2 points 2 days ago

This could further accelerate the arms race between malicious srappers and websites.

My fear is this would create collateral damage, block legitimate scrappers and visitors, hassle people with an increasing number of captcha.

[-] TerHu@lemmy.dbzer0.com 3 points 2 days ago

yeah i think that there’s a good chance for vpn users to be harassed by anti ai measures

[-] dis_honestfamiliar@lemmy.sdf.org 1 points 2 days ago

They probably can. Link

this post was submitted on 01 Jul 2025

127 points (99.2% liked)

Privacy

39550 readers

545 users here now

A place to discuss privacy and freedom in the digital world.

Privacy has become a very important issue in modern society, with companies and governments constantly abusing their power, more and more people are waking up to the importance of digital privacy.

In this community everyone is welcome to post links and discuss topics related to privacy.

Some Rules

Posting a link to a website containing tracking isn't great, if contents of the website are behind a paywall maybe copy them into the post
Don't promote proprietary software
Try to keep things on topic
If you have a question, please try searching for previous discussions, maybe it has already been answered
Reposts are fine, but should have at least a couple of weeks in between so that the post can reach a new audience
Be nice :)

Related communities

much thanks to @gary_host_laptop for the logo design :)

founded 5 years ago

MODERATORS