Most advice about AI bots starts with a single assumption: that you want to keep them out. Add a Disallow to robots.txt, drop in a firewall rule, install a plugin, and you’ve solved the problem.
That framing was right for 2024. In 2026 it’s half the picture — and increasingly the less important half.
There are now over 60 distinct AI bot user agents crawling the web, and they don’t all want the same thing. Some collect training data and give you nothing back. Others fetch a page because a real person just asked ChatGPT or Perplexity a question — and then cite you, sending a high-intent visitor to your site. Blocking the first group makes sense. Blocking the second group means disappearing from the place your future customers are increasingly making decisions.
So the real question isn’t “how do I block AI bots?” It’s “which bots do I block, which do I want reading me, and how do I tell the difference?” This guide covers both halves: the full toolkit for controlling AI crawlers on WordPress, and the bigger opportunity most site owners are missing.
First, Why robots.txt Isn’t Enough
If you’ve added Disallow rules for AI crawlers, you might assume the problem is handled. It isn’t.
robots.txt was designed in 1994 as a voluntary protocol. It relies on crawlers choosing to read and respect it. Search engines complied for 30 years because violating robots.txt could get them blocked from the broader web. AI crawlers operate under different incentives — many don’t need to stay in a webmaster’s good graces, and based on detection data from AI Bot Tracker installations, roughly 13% of known AI bots have been observed ignoring robots.txt directives entirely.
robots.txt has no enforcement mechanism. It’s a text file that says “please don’t crawl this.” A “No Soliciting” sign is a suggestion; a locked door is enforcement. robots.txt is the sign, not the lock.
That’s the hook that sends most people down the blocking path. So let’s walk the whole path — and then talk about why the destination isn’t where you think.
The Real Costs That Push People to Block
Blocking is a reaction to real costs. Three of them:
Bandwidth and server load. Every AI bot request costs bandwidth. A blog post is 80–150 KB of HTML; multiply that across hundreds of posts and dozens of bots re-crawling monthly and it becomes a measurable line item. Worse is server load: every WordPress request consumes a PHP worker and database queries. Aggressive crawlers like Bytespider — which can hit a single site hundreds of times a day — compete with your real visitors for a shared pool of 2–4 PHP workers on typical hosting. Standard analytics won’t show you any of this, because Google Analytics, Plausible, and Fathom all filter bots out by design. You’re paying for traffic you can’t see.
Content value. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect your content to improve a model. It’s a one-way extraction — your work goes in, and you get no traffic, citation, or compensation back.
Loss of control. New crawlers appear constantly, and some companies run secondary crawlers under different user-agent names that your blocklist doesn’t cover. You’re always playing catch-up.
Those costs are real. But notice what they have in common: you can’t weigh any of them until you can see what’s actually hitting your site.
The Full Toolkit: 5 Ways to Control AI Bots
Here’s every method, from weakest to strongest enforcement.
1. robots.txt (baseline)
User-agent: GPTBot
Disallow: /
User-agent: Bytespider
Disallow: /Fine as a first layer for compliant bots. Voluntary, requires you to know every user-agent, and ~13% ignore it. Never your only defense.
2. Server-level rules (.htaccess / Nginx)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|Bytespider) [NC]
RewriteRule .* - [F,L]Enforceable because the server refuses the request — the bot’s cooperation isn’t required. Downsides: Apache-only syntax, manual user-agent lists, sophisticated bots spoof their user-agent, no analytics (blocked requests vanish silently), and a syntax mistake can take down your whole site.
3. Cloudflare / CDN rules
If you’re behind Cloudflare, you can block by user-agent or ASN at the network edge — including a free “AI Scrapers and Crawlers” toggle. Effective, but adds infrastructure complexity, and overly-aggressive rules can catch legitimate traffic.
Watch out: your host may already be doing this without telling you. Several managed WordPress hosts (WP Engine, SiteGround) block AI crawlers at the Cloudflare/WAF layer before the request ever reaches WordPress — with no self-service opt-out. If your bot dashboard shows suspiciously few AI visits, infrastructure-level filtering upstream is the likely cause, and it may be quietly costing you AI-search visibility on bots you’d actually want to allow. Kinsta and Pressable leave the decision to you.
4. WordPress plugin detection + response strategies
A plugin operates at the PHP layer with full visibility and a dashboard. This is where you move from blind blocking to informed control. Once a bot is detected, you have six response strategies — each for a different intent:
| Strategy | What it does | Best for |
|---|---|---|
| Log Only | Records the visit; bot gets normal content | Monitoring, and bots you want (Perplexity, Applebot) |
| Block 403 | Returns Forbidden, no content | Clear denial of unwanted bots |
| Tarpit | Drips the response over minutes/hours, tying up the bot’s connection | Aggressive, high-volume crawlers |
| Rate Limit 429 | ”Slow down” with a Retry-After header | Diplomatic throttling of bots you tolerate |
| Decoy Content | Serves plausible fake content | Poisoning training-data collection |
| Shadowban | 200 OK with an empty body — bot thinks it succeeded | Stealth denial of adaptive crawlers |
Most sites end up using two or three: Log Only for allowed bots, Block 403 or Shadowban as the default, and Tarpit for the worst offenders.
5. Honeypot detection (catches the ones that hide)
The methods above rely on a bot honestly declaring itself. The most dangerous crawlers don’t — they wear a standard Chrome or Firefox user-agent. Honeypots catch them by behavior.
The technique: inject a link that’s invisible to humans (hidden with CSS) but present in the raw HTML. Humans never see it, so they never follow it. A bot parsing every link does follow it — and reveals itself. The false-positive rate is effectively zero, because no human can click a link they can’t see. It’s the only method that reliably catches disguised crawlers and bots ignoring robots.txt.
Policy layer: ai.txt and llms.txt
Two emerging standards go beyond robots.txt’s crude allow/deny. ai.txt declares per-bot permissions for training, summarization, and attribution. llms.txt is a curated content guide for AI systems. They’re declarations, not enforcement — but they’re the bridge to the second half of this article, because llms.txt isn’t about keeping bots out. It’s about helping the right ones find your best content.
The Plot Twist: Blocking Everything Costs You Customers
Here’s the shift almost nobody built their bot strategy around.
Not all AI bots are extractive. They split into two camps:
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) take your content to train a model. No traffic back. Block these freely if content protection is your goal.
- Search and retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch a page because a person just asked an AI assistant a question — and then cite you. That’s a referral, with attribution, from someone actively making a decision.
And that referral traffic converts. Multiple 2026 analyses found AI-referred visitors convert at roughly 4.4x the rate of standard organic search traffic, because they arrive having just gotten a direct recommendation from an assistant. Cloudflare reported in 2026 that more than half of all web requests now come from machines, not humans — the agents are the new front door.
So a blanket block does something worse than protect you: it makes you invisible at the exact moment a decision gets made. People increasingly ask an assistant instead of scrolling a results page. If your content isn’t readable and citable by that assistant, you don’t rank low — you don’t exist.
The Bigger Play: Get Cited, Not Just Protected
This is the discipline the industry is racing to define: Answer Engine Optimization (AEO) — optimizing your content to be surfaced and cited by ChatGPT, Claude, Perplexity, and Google’s AI answers, instead of (or alongside) ranking in blue links.
The move isn’t “block or allow.” It’s deliberate, per-bot triage:
- Block the extractors you get nothing from — training crawlers, aggressive bandwidth hogs — using the toolkit above.
- Actively welcome the citation engines — the search/retrieval bots that send referral traffic — and make their job easy with clean content and an llms.txt guide pointing at your best pages.
- Verify it’s actually working — because you can’t manage what you can’t measure.
That third point is the catch. You cannot make any of these decisions blind. You need to know which bots visit, how often, which pages they read, and whether the citation engines you want are actually reaching you (or being silently filtered by your host). Visibility isn’t step one of blocking — it’s step one of the entire strategy, offensive and defensive.
Start With Visibility
Whether your goal is to block extractive crawlers, get cited by the right ones, or both, the first move is the same: see what’s actually happening.
AI Bot Tracker is a free WordPress plugin that makes AI crawler traffic visible — the traffic your analytics hides. Install it, and within 24–48 hours your dashboard shows every AI bot that visited, how often, which pages they targeted, and — via honeypot detection — which ones ignored your rules. Most sites discover 5–15 distinct AI bots in the first week.
From there the decisions are yours to make, deliberately rather than by default:
- Track which bots are actually scraping your content — the visibility foundation for everything else.
- Reference the full 2026 AI bot list — who operates each crawler, what it’s for, and whether it respects robots.txt.
- Declare policy with ai.txt and llms.txt — the standards that also help the right AI systems find and cite you.
- Go deeper on getting cited — the full AEO playbook for making your WordPress site visible to the agents your customers now ask.
Blocking AI bots is a legitimate goal, and the tools to do it well are all here. But in 2026, the sites that win aren’t the ones with the highest walls. They’re the ones that block what drains them, welcome what cites them, and — crucially — can see the difference.
Install AI Bot Tracker (free) and start with visibility.