· 10 min read · Crawlantix

Cloudflare Blocks AI Crawlers: Which Still Reach You?

Cloudflare's AI crawler settings changed on Sept 15. Here's how to check which AI bots still reach your WordPress site and stay cited in AI search.

How do you check which AI bots can still reach your WordPress site after Cloudflare’s September 15 change? Confirm whether your site is proxied through Cloudflare, read your AI bot policy settings, then compare your origin access logs against each crawler’s published user agent and IP list. Requests Cloudflare blocks never reach WordPress, so your origin logs show only what actually got through.

If you want to be cited in ChatGPT search, Claude or Perplexity, the useful question is less “am I blocking AI crawlers?” and more “am I accidentally blocking the ones that send people to me?”

What changed on September 15?

In the bot management changes it announced for “Content Independence Day” on July 1, Cloudflare started sorting AI traffic into three behaviors: Search (indexing content to answer questions later), Agent (acting in real time on a person’s behalf) and Training (collecting content to train or fine-tune a model). Each behavior can be set to Allow, Block on all pages, or Block on pages with ads, under Security Settings > Configure AI bot policies (Cloudflare docs).

Cloudflare announced on July 1 that new domains onboarding from September 15, 2026 would get updated defaults. On September 15 it published a refinement that supersedes parts of the July plan: the legacy “Block AI Bots” toggle is deprecated, Managed robots.txt is being replaced by “Bot Preference Sync”, and a new Disallow AI Training option was added (Cloudflare blog, Sept 15). Existing domains were migrated from their old settings rather than reset. Cloudflare’s Pay Per Crawl and Content Signals sit beside these settings: one charges crawlers at the edge, the other only states a preference in robots.txt.

Who does the new default apply to?

Cloudflare’s own changelog describes the change as defaults for new domains onboarding to Cloudflare, and says all customers could opt out before September 15. Some press coverage reported a broader scope. TechCrunch wrote that the blocking would apply to new customers, new sites set up by existing customers, and all existing free customers.

Because the two descriptions differ, don’t assume either one applies to you. Open your dashboard and read what your zone is set to today. Cloudflare says existing domains that had only the old “Block AI bots” toggle on were migrated to Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent. Domains that already used the granular controls kept their values, except that a Training setting of Block or Block on pages with ads became Disallow AI Training.

Why does this affect your AI visibility?

These settings decide which AI products can read your pages, and some of those products send you visitors. OpenAI says OAI-SearchBot exists “to surface websites in search results in ChatGPT’s search features.” Anthropic says blocking Claude-SearchBot may reduce your site’s visibility in user search results.

The presets for new ad-supported domains are Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads. For domains without ads, all three are set to Allow. So on an ad-supported WordPress blog, an assistant fetching a page because a reader asked about it may now be refused on pages that show ads. That can be the right call, as long as you made it on purpose. Our guide to whether you should block AI bots at all walks through the trade-off.

What happens to Googlebot, Bingbot and Applebot?

These are mixed-use crawlers: one bot serves both search and AI training. Cloudflare labels Apple, Google and Microsoft “Accountable” because they offer, or have committed to offering, a training opt-out that leaves traditional search alone.

Disallow AI Training publishes a robots.txt preference instead of blocking these bots, so they keep crawling for search. Google says Google-Extended “does not impact a site’s inclusion in Google Search” and has no separate user agent string (Google). Apple uses Applebot-Extended. Bing is the exception for now. Cloudflare’s Sept 15 post says Microsoft is building robots.txt support for a no-training preference, targeting early 2027, and until then this setting won’t pass that preference to Bing.

Choosing Block for Training works differently. Cloudflare states it “will stop Applebot, Bingbot, and Googlebot from reaching your site,” search included. Check which option your Training row actually uses.

Which AI crawlers should you look for?

OpenAI and Anthropic run separate search, user-triggered and training crawlers, and Perplexity runs separate search and user-triggered crawlers, which is what makes per-category decisions possible. Cloudflare notes that Amazon, Anthropic, Meta and OpenAI separate search from training, so training can be refused without touching search. For a longer directory, see our complete list of AI bots for 2026.

Crawler Operator Stated purpose Expected under the ad-supported new-domain preset* How to verify
OAI-SearchBotOpenAISearch (ChatGPT search results)Allowedopenai.com/searchbot.json
ChatGPT-UserOpenAIUser actions in ChatGPT and Custom GPTsBlocked on pages with ads if classed as Agentopenai.com/chatgpt-user.json
GPTBotOpenAITraining foundation modelsBlocked (Disallow AI Training)openai.com/gptbot.json
Claude-SearchBotAnthropicSearch result qualityAllowedclaude.com/crawling/bots.json
Claude-UserAnthropicFetches pages when users ask ClaudeBlocked on pages with ads if classed as Agentclaude.com/crawling/bots.json
ClaudeBotAnthropicTraining data collectionBlocked (Disallow AI Training)claude.com/crawling/bots.json
PerplexityBotPerplexitySurfacing sites in Perplexity searchAllowedperplexity.com/perplexitybot.json
Perplexity-UserPerplexityUser-requested page visitsBlocked on pages with ads if classed as Agentperplexity.com/perplexity-user.json
GooglebotGoogleSearch crawling (training opt-out via Google-Extended token)Allowed; training preference published in robots.txtReverse plus forward DNS, or Google's IP JSON files

*Which bucket each user-triggered bot falls into is Cloudflare’s classification, so confirm it in your own dashboard. Purposes and IP lists come from each operator’s crawler documentation, listed in References.

How do you check which AI bots still reach your site?

Work from the outside in.

  1. Check whether the hostname is proxied. An orange-cloud (Proxied) DNS record routes traffic through Cloudflare. A grey-cloud (DNS-only) record goes straight to your origin, and Cloudflare’s protections don’t apply. From a terminal, curl -sI https://yoursite.com returns a cf-ray header when proxied.
  2. Read your AI bot policy. Note the Search, Agent and Training values, and whether Training says Block or Disallow AI Training.
  3. Open AI Crawl Control. Cloudflare says it is available on all plans. It shows allowed and unsuccessful requests per crawler, where “unsuccessful” covers blocks and other errors (Cloudflare docs). This is the edge view.
  4. Grep your origin access log. For example: grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log. This is the origin view.
  5. Verify source IPs. Match them against each operator’s published list. Behind Cloudflare, log the CF-Connecting-IP header so you see the real client IP.
  6. Compare the two views. A crawler you want that shows at the edge but not at the origin is usually being stopped by a Cloudflare rule. Check the status codes in AI Crawl Control to confirm.

How do you tell Cloudflare-blocked from not-proxied?

Use the cf-ray result from step 1 as the switch. With no cf-ray header, Cloudflare isn’t in the request path for that hostname, and the September 15 defaults change nothing for you. Any AI crawler blocking on that site comes from robots.txt, a security plugin, your host’s firewall, or nothing at all.

If the site is proxied, a bot Cloudflare refuses is answered at the edge and never reaches WordPress. It won’t appear in your origin log or in any plugin that records visits inside WordPress. So a bot’s absence from your origin log is ambiguous on its own. Pair it with AI Crawl Control: present at the edge and absent at the origin usually means a Cloudflare rule stopped it, while absent from both usually means the crawler hasn’t visited. There is one caveat. “Block on pages with ads” relies on Cloudflare having seen ads on a page, so a page it hasn’t classified yet may still let requests through.

How should you decide per category?

Start from what you want AI products to do with your content, then set each row to match. Our AI crawler visibility and AEO guide goes deeper on getting cited.

Search: If you want citations and referral clicks from AI search, keep Search allowed and confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot show up at your origin.

Agent: User-triggered fetchers arrive because a person asked about your page. Blocking them on ad pages protects ad inventory but can stop an assistant from quoting you. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, so if you do decide the lost citations are worth it, an edge rule is the more dependable control.

Training: Disallow AI Training refuses training-only crawlers and keeps Googlebot and Applebot crawling for search. For many publishers it’s a sensible middle setting.

If you’d like an ongoing record instead of a one-off grep, Crawlantix’s AI Bot Tracker plugin logs AI bot visits at the WordPress origin. Here’s how to track AI bots scraping your content.

FAQ

Did Cloudflare block AI crawlers on my existing site?

Not by resetting it. Cloudflare’s docs describe new defaults for new domains and say existing domains were migrated from their prior settings. Some coverage reported a broader free-plan scope, so open Security Settings and read your current Search, Agent and Training values.

How do I check which AI bots visit my site?

Grep your server access log for AI user agents such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then verify the IPs against each operator’s published list. If you use Cloudflare, compare with AI Crawl Control to see requests refused before they reached WordPress.

Not if you use Disallow AI Training. It publishes a robots.txt preference and keeps Googlebot crawling for search, and Google says Google-Extended doesn’t affect Search inclusion. Choosing Block for Training is different: Cloudflare says it stops Googlebot, search included.

Why don’t blocked bots show up in my server logs?

When your site is proxied, Cloudflare answers blocked requests at its edge, so they never reach your origin server. Your WordPress logs only show traffic that passed through. Use AI Crawl Control for the edge view and your origin logs for what actually arrived.

Can a fake bot use GPTBot’s user agent?

Yes. Any client can send any user agent string, so a name match proves nothing on its own. OpenAI, Anthropic and Perplexity publish IP lists for their crawlers, and Google documents reverse plus forward DNS verification. Match the source IP before trusting the label.

References

Try AI Bot Tracker — Free on WordPress.org

Detect, monitor, and respond to AI crawlers on your WordPress site. Full bot detection is free forever.

Download Free Plugin