46

Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content.

Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:

  1. Allow: Grant the crawler free access to content.
  2. Charge: Require payment at the configured, domain-wide price.
  3. Block: Deny access entirely, with no option to pay.

Although this is old news, I still found it interesting. Also, I like the "AI Labyrinth" feature of Cloudflare to block AI bots.

you are viewing a single comment's thread
view the rest of the comments
[-] cockmushroom@reddthat.com 3 points 3 days ago

what if youlre not trying to train ai? What if you just want to update a personal dashboard/archive?

[-] Hippy@piefed.social 3 points 3 days ago

Well, cockmushroom, then you're building a pretty expensive dashboard. I still don't see how they are going to differentiate between normal browsing and bot crawling. I don't think that's an exact science.

this post was submitted on 09 Feb 2026
46 points (96.0% liked)

Web Development

5450 readers
1 users here now

Welcome to the web development community! This is a place to post, discuss, get help about, etc. anything related to web development

What is web development?

Web development is the process of creating websites or web applications

Rules/Guidelines

Related Communities

Wormhole

Some webdev blogsNot sure what to post in here? Want some web development related things to read?

Heres a couple blogs that have web development related content

CreditsIcon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 2 years ago
MODERATORS