Pay Per Crawl vs Pay Per Use for Small Sites

Pay Per Crawl and Cloudflare's Pay Per Use beta explained for small sites, and why you need your own AI bot logs to price or audit AI access.

Can a small WordPress site actually get paid by AI crawlers through Cloudflare’s Pay Per Crawl? Possibly, but nobody has published payout data yet, so treat any earnings claim as unproven. What you can do today is measure which AI bots fetch which of your pages, because without that record you have nothing to price, negotiate or audit.

Cloudflare now runs two programs side by side. Pay Per Crawl charges for access at the edge. Pay Per Use, in beta since September 30, 2026, pays when a buyer reports that it used your content. Both depend on numbers produced by Cloudflare or the AI companies buying access, not by you. This post covers how each one works, which other licensing schemes exist, and the first-party data a small site owner should be keeping before signing up for any of them.

How does Pay Per Crawl actually work?

Pay Per Crawl puts a price check in front of your web content, and AI web crawlers either agree to pay or get refused. Cloudflare launched it in private beta on July 1, 2025 as a “third path” between blocking and free access.

A crawler that hasn’t signaled payment intent gets HTTP 402 Payment Required with a crawler-price header. It can retry with crawler-exact-price, or send crawler-max-price up front, and a successful charge comes back with crawler-charged. Cloudflare acts as Merchant of Record. For each crawler you pick Allow, Charge or Block, and “Charge” against a crawler with no billing relationship behaves like a 403.

To stop a web scraper from posing as a paying bot, Cloudflare uses Web Bot Auth to sign requests: Ed25519 keys, a public key directory and HTTP Message Signatures. Signed requests are what let a 402 mean something, since a user agent string alone proves nothing. As of Cloudflare’s July 28, 2026 docs, Pay Per Crawl is still a closed beta inside AI Crawl Control, joined by signup form or, on Enterprise, through your account executive.

What changed with Pay Per Use?

Pay Per Use pays for what an AI product does with a page after fetching it, instead of charging for the fetch. The September 30 beta post puts it in one line: “Pay Per Use pays for what happens next.”

The pricing model is flipped compared with Pay Per Crawl. The buyer defines the paid use and sets the price. You accept or decline under Dashboard > Monetize > Pay Per Use, with no origin change. Program terms set what the buyer may do with the content, including training restrictions, and payouts are monthly.

Two things to clear up. First, Pay Per Use does not replace Pay Per Crawl; Cloudflare says publishers “can choose the model that suits them.” Second, Ceramic.ai and You.com were named on July 1, 2026 as experiment partners, and the beta post names no buyers. The beta post itself lists no eligibility criteria and no general availability date, and Search Engine Journal notes that neither of Cloudflare’s September 30 posts reports transaction counts or payouts.

Who reports the usage you get paid for?

The buyer does. Under Pay Per Use, each buyer sends one JSON line per use (timestamp, URL, event ID) to Cloudflare. Cloudflare’s post says the usage is self-reported, and that Cloudflare only checks that each reported use maps to an enrolled publisher.

That moves the audit job onto you. If a buyer’s monthly report lists uses of, say, /guides/sourdough-starter/, the only independent evidence you can hold up against that line is your own record of which bots fetched that URL, when, and how often. Cloudflare’s Business Insights “shows which crawlers visit,” which helps, but that is still Cloudflare’s view of its own edge.

So the useful question for a small site is less “how much will I earn?” and more “will I be able to tell whether the report is plausible?” A fetch is not a use, so your logs can’t prove a use happened. They can show a buyer reporting uses for pages its bot never requested.

What other AI licensing programs exist?

Several, from RSL’s License: line in robots.txt to TollBit’s bot paywall subdomain. The columns that matter here are who sets terms and who counts usage. Prices and revenue shares are left out on purpose: no program has published small-site payouts. For a small site, these are an alternative to bilateral licensing deals, which are hard to negotiate one AI company at a time.

Program Who sets terms Who reports usage Needs Cloudflare? Who can join
Cloudflare Pay Per CrawlPublisher sets price per crawlCharges happen at the edge, per requestYesClosed beta, signup form
Cloudflare Pay Per UseBuyer defines use and price; publisher accepts or declinesBuyer, self-reported per useYes, runs in the dashboardBeta; eligibility not stated
RSL 1.0Publisher, via an XML license linked from robots.txtDepends on the CDN or licensorNo (Cloudflare, Akamai, Fastly support it)Any site can add a template
TollBitPublisher, via a bot paywall subdomainTollBitNo; redirects bots to a TollBit subdomainPublisher signup on TollBit's site
Microsoft PCMPublisher defines licensing and usage termsMicrosoft, usage-basedNoPilot, announced Feb 3, 2026
Perplexity Comet PlusPerplexityPerplexity (visits, citations, agent actions)NoApply by email

RSL’s own docs list “All content publishers” for its collective template, though The Verge notes RSL alone can’t block scrapers that don’t pay. Machine-readable policy files like this sit in the same family as the ai.txt and llms.txt standards.

Why can’t you price AI access without your own data?

Because the value of a bot visit depends on which bot it was and what it sends back, and only your logs show that for your site. Cloudflare’s crawl-to-refer ratio makes the point at internet scale: HTML requests from a platform’s crawlers divided by requests that arrive with that platform as the Referer.

Cloudflare’s August 29, 2025 post put Anthropic at roughly 38,000:1 in July 2025, down from nearly 287,000:1 in January, and Perplexity at about 195:1. Training accounted for nearly 80% of AI bot activity. Readings for 2026 come from third parties querying Radar data; SEOmator’s reading of September 2026 data (pulled October 7), for example, has Anthropic at 549:1 and Google at 4.7:1. Figures vary by date range and method.

Cloudflare also notes that Claude’s native app, and likely others, send no Referer, so these ratios “may overstate” the imbalance by an unclear amount. Your own ratio will differ from the global one. If you want to set a Pay Per Crawl price, or judge whether a Pay Per Use offer is fair, you need it.

Training, search and user fetch are three different deals

A training crawl for large language models, a search index crawl and a user-triggered fetch by AI agents are worth different things to you, and the major operators label them separately. OpenAI runs GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for user-initiated fetches, where it says “robots.txt rules may not apply.” Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way. Perplexity says PerplexityBot is “not used to crawl content for AI foundation models,” and that Perplexity-User generally ignores robots.txt.

Google-Extended trips people up. It is a robots.txt token with no user agent of its own, so you will never see it in a log. Crawling happens under existing Google user agents, and Google says it does not affect inclusion in Google Search.

A search bot that sends readers back may be worth leaving free, since blocking it can keep you out of ChatGPT search and Perplexity answers. A training crawler that never refers anyone is the one to charge or block outright. The complete list of AI bots for 2026 has the long tail.

What should a small site owner measure now?

Build the record you’d need to check a buyer’s report first.

  1. Find out if you’re proxied. curl -sI https://yoursite.com returns a cf-ray header when Cloudflare is in the path. Our post-September 15 Cloudflare check walks through the edge settings.
  2. Read your origin log by user agent. grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log, then verify IPs against each operator’s published list, because user agents are easy to fake.
  3. Bucket each hit as training, search or user fetch. Use the operator’s documented purpose, not a guess from the name.
  4. Compare crawls to AI referrals. Count visits arriving with chatgpt.com, claude.ai or perplexity.ai as the Referer, per page, against fetches by that operator’s bots.
  5. Keep a per-page history. Buyers report each use, and Cloudflare pays out monthly. You want months of fetch counts per URL to line up against those reported uses.

If you’d rather not run greps on a schedule, Crawlantix’s free AI Bot Tracker plugin logs AI bot visits at the WordPress origin, including which pages each bot read. Here’s how it works. It records what reached WordPress, so pair it with Cloudflare’s edge view if you’re proxied.

FAQ

Can a small WordPress site use Pay Per Crawl?

Only if the site runs through Cloudflare, and Pay Per Crawl is still a closed beta you request through a signup form. Cloudflare’s docs don’t state a plan-tier requirement. To confirm your site is proxied before you apply, look for a cf-ray header in its HTTP responses.

Is Pay Per Use replacing Pay Per Crawl?

No. Cloudflare says publishers can choose the model that suits them. Pay Per Crawl charges each fetch at the edge. Pay Per Use pays monthly for uses a buyer reports, with the buyer setting the price and the publisher accepting or declining it in the dashboard.

How much do sites earn from AI content licensing?

There is no public answer. Cloudflare’s Pay Per Use beta post gives no payout figures, and none of the other programs have published small-site earnings. Be skeptical of any guide that quotes a number for your site without having seen your traffic.

How do I check a buyer’s Pay Per Use report?

Match each reported URL against your own log of that buyer’s bot fetching it. Usage is self-reported, and Cloudflare only checks that the use maps to an enrolled publisher. A fetch doesn’t prove a use, but reported uses on never-fetched pages are worth questioning.

Does blocking Google-Extended remove my site from Google?

No. Google says Google-Extended doesn’t affect Search inclusion or ranking. It is a robots.txt token without its own user agent, and it controls whether content is used for Gemini training and grounding. Googlebot keeps crawling and indexing your pages for Search as before.

References

Try AI Bot Tracker — Free on WordPress.org

Detect, monitor, and respond to AI crawlers on your WordPress site. Full bot detection is free forever.

Download Free Plugin