For most of the AI-crawler era, two things were true at once. Bots declared who they were with a user-agent string anyone could type. And site owners decided who to allow by matching that string against a blocklist they had to maintain by hand. Both halves ran on trust, and neither was verifiable.
That arrangement is now being rebuilt in the open. The infrastructure companies that sit in front of a large share of the web have begun shipping a real declaration-and-verification layer: a way for an AI agent to prove it is who it claims to be, state what it does with what it reads, and be listed somewhere a site owner can check. This is a genuinely new layer, distinct from the ai.txt and llms.txt files you publish and the blocklists you maintain. It changes the question site owners have been asking.
The old question was “which user-agent strings do I block?” The new one is “who has actually declared and proven themselves, and do I agree with what they say they’ll do with my content?” This post is about that shift, and what to do now while it settles.
Why the Honor System Ran Out of Road
The Robots Exclusion Protocol, standardized as RFC 9309, is a voluntary agreement. A crawler reads your robots.txt and chooses whether to obey it. That worked for search engines because they needed to stay in webmasters’ good graces to keep indexing the web.
AI crawlers stressed that model in two ways. First, a user-agent header is trivially spoofable. Anything can send a request that says GPTBot or ClaudeBot in the header, and your robots.txt rule has no way to know whether the request is really from OpenAI or Anthropic or from a scraper wearing their name. Second, the number of distinct agents kept climbing, so keeping a hand-maintained blocklist accurate became a losing game. And plenty of crawlers simply opt out of the honor system: based on detection data from AI Bot Tracker installations, roughly 13% of known AI bots have been observed ignoring robots.txt directives entirely. We cover the full roster in our complete list of AI bots in 2026, and the honest takeaway is that the list is never done.
So the ecosystem is doing what it usually does when an honor system stops scaling: it is adding identity you can verify and a place to look people up.
The Three Moving Parts of the New Layer
Three things are being built at once, and they fit together. It helps to see them as one layer rather than three separate news items.
1. Cryptographic identity: Web Bot Auth
The biggest change is that a bot can now cryptographically sign its requests. Web Bot Auth is an approach where an agent signs each HTTP request with a private key, and the receiving site verifies that signature against the operator’s published public key. It builds on the IETF’s HTTP Message Signatures work, carried in the Signature and Signature-Input headers.
Google documents its own implementation of Web Bot Auth and describes it as an experimental cryptographic protocol used to authenticate requests sent by bots, verified by following the HTTP Message Signatures standard. Experimental is the right word to keep in mind. This is early. But the direction is clear: identity is moving from an easily-forged header to a signature that is expensive to fake, and it decouples a bot’s identity from the IP range it happens to crawl from.
For a site owner, the practical promise is simple. A signed request from a citation engine you want, like the retrieval fetchers behind ChatGPT, Claude, and Google’s Gemini, can eventually be told apart from something merely pretending to be one. That is a foundation the old user-agent match never had.
2. A directory: Cloudflare’s BotBase for operators
Identity is only half of it. You also need somewhere to look a bot up. Cloudflare has been building BotBase, a directory of known bots and agents, and it recently opened the operator side so the companies running crawlers can submit and maintain their own entries rather than waiting on a form with no reply.
What is interesting for site owners is what operators are asked to declare. Per Cloudflare’s own description, an operator states three things: what the bot does (indexing, acting for a user, data collection, model training), how it uses what it reads, and who is actually running it. Submissions are checked against verification methods including IP lists, reverse DNS, and Web Bot Auth signatures. In other words, the directory and the cryptographic identity are wired together. A declaration is worth more when the directory can confirm the declarer is real.
You do not have to use Cloudflare to benefit from the pattern. The pattern itself is the point: declared behavior, declared content use, and a verification method, in one lookup.
3. A content-use vocabulary: Content Signals in robots.txt
The third piece answers a question robots.txt could never express: not just “may you crawl this,” but “what may you do with it once you have it.” Cloudflare’s Content Signals approach adds machine-readable signals to robots.txt for exactly that, expressing preferences like whether content may be used for search, as input to an AI answer, or to train a model, with each set to yes or no.
This is the same instinct behind the ai.txt policy file we covered earlier, pushed into the file crawlers already read. It closes the gap between “I told you not to crawl” and “I told you what you may do with my content,” which are very different statements. A retrieval bot summarizing your page to cite you and a training crawler absorbing it into a model are doing different things, and for the first time you have vocabulary to treat them differently in the file every compliant crawler already checks.
The Crawlantix Point of View
This is the part the news-roundup version misses, and it is what matters most for a site owner.
Declaration and verification are not the same as compliance, and neither is the same as reality on your server. Web Bot Auth proves a bot is who it says. A directory records what a bot says it does. Content Signals record what you want done with your content. All three are statements of intent and identity. None of them tell you what actually hit your site last night.
That gap is the whole reason this layer needs a fourth part that the infrastructure vendors cannot provide for you: your own ground truth. A bot can be verified and still crawl pages you would not expect. A well-behaved citation engine can be silently filtered out by your host before it ever reaches WordPress. An unsigned, undeclared scraper can still walk in wearing a familiar name. The declaration layer raises the floor. It does not close the loop.
So our view is not “verification is hype” and not “verification solves it.” It is this: the new layer finally gives you a credible way to decide who gets to crawl you, but you still have to watch what they actually do to know whether your decision is holding. Declaration sets policy. Measurement confirms it. You need both, and only one of them is being handed to you by the platforms.
This also reframes the per-bot triage we keep coming back to. The move was never “block or allow.” It is block the extractors you get nothing from, welcome the citation engines that send referral traffic, and measure which are actually reaching you. The declaration layer makes the first two decisions more trustworthy, because now the citation engines you want to welcome can prove they are the citation engines. The measurement half is still yours to own.
What to Do Now (WordPress, Practically)
You do not need to wait for any of this to finish standardizing. There are useful moves today, in rough order of effort.
-
Add content-use signals to your robots.txt. Alongside your existing rules, state your preference for search, AI input, and AI training. It is a small, machine-readable declaration that costs minutes and positions you for the tools now reading it. Keep your existing disallow rules in place; this is additive.
-
Keep publishing your llms.txt. The curated content guide is still the cleanest way to point the retrieval bots you want at your best pages. Declaration files and identity verification are complementary, not competing.
-
Decide your stance per behavior, not per name. The directory model rewards owners who think in terms of what a bot does with content rather than which brand it belongs to. Training extraction, live retrieval for citation, and background indexing are three different value exchanges. Write your preferences accordingly.
-
Treat verification as a floor, then measure. As signed and directory-listed bots become common, they make your allow decisions safer. But the only way to know whether a verified bot behaves as declared, or whether a citation engine you welcomed is actually reaching you, is to see the AI crawl traffic your analytics hides.
Where Crawlantix Sits in This
This is exactly the vantage point Crawlantix is built for. The declaration layer is telling you what bots say. The AI Bot Tracker plugin shows you what they do: which AI agents actually visited, how often, and which pages they read, on your own server, in traffic standard analytics filters out.
That is the check the new layer cannot do for itself. When a bot is verified and directory-listed and honoring your content signals, you want to confirm its behavior matches its declaration. When a citation engine you deliberately welcomed goes quiet, you want to catch it. Declaration is the policy. Your crawl log is the evidence.
The web is finally building a way to answer “who gets to crawl you.” The missing half is proof that the answer is holding, and that is measured on your site, not in someone else’s directory.
Install AI Bot Tracker (free) and see who is actually crawling you, verified or not.