Explainer
Cloudflare Verified Bots: How a Crawler Gets Recognized
What Cloudflare's verified bots are, the rules a crawler must meet, the three ways to prove who it is (Web Bot Auth, IP lists, reverse DNS) and how to apply.
By ZeroCaptcha Engineering5 min readPublished
A Cloudflare verified bot is “a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does”: search engine crawlers, monitoring services and user-driven agents, for example. To be verified, a bot must meet two standards: honest self-identification (“through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS”) and non-abusive behaviour (it “obeys robots.txt and crawl directives, maintains reasonable request rates, and has not been observed evading website owner preferences or attacking sites”). Site owners can then allow or block verified bots by category in their rules. The operator applies through a form in the Cloudflare dashboard.
This explainer covers the requirements, the three verification methods, what site owners see, and what it means for an ordinary crawler, as Cloudflare documented it on 1 October 2026.
Why it matters to a crawler
A site’s rules often treat verified bots differently from everything else. Cloudflare gives site
owners two fields for that: cf.bot_management.verified_bot, true for a verified bot, and
cf.verified_bot_category, its category. Cloudflare’s own example rule for JavaScript detections
includes not cf.bot_management.verified_bot, so that verified bots are not challenged by it.
A crawler that qualifies and gets verified can be let through where an unknown client meets a
challenge page; one that doesn’t is judged like any other traffic.
The requirements
| Requirement | What Cloudflare asks |
|---|---|
| Honest self-identification | It “declares who it is deterministically”: a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS |
| Non-abusive behaviour | It “obeys robots.txt and crawl directives, maintains reasonable request rates, and has not been observed evading website owner preferences or attacking sites” |
| A specific user agent | Generic ones are rejected: Cloudflare names Dart, Go-http-client, GuzzleHttp, Chrome, Firefox, Safari, Nessus, node and python-requests among them |
| Exclusive addresses | Published IPs must be used by the bot owner only |
A verified bot can lose its status. Cloudflare’s examples of policy breaches include adding addresses it does not use exclusively, undisclosed addresses, traffic that contradicts the stated purpose, and “An AI Crawler that does not respect the crawl-delay directive in robots.txt.”
The three ways to prove who you are
| Method | How it works | What you publish |
|---|---|---|
| Web Bot Auth | Each request is signed with an Ed25519 key; Cloudflare checks the signature against your public key | A key directory at /.well-known/http-message-signatures-directory |
| Public IP list | Cloudflare fetches your list and matches requests’ source addresses | A fixed, limited set of IP addresses in plain text, JSON or CSV, plus a user-agent pattern |
| Reverse DNS | Cloudflare looks up the PTR record of each request’s address and checks it ends in your domain | Domain suffixes, PTR records, and a user-agent pattern |
Web Bot Auth is the newest. Cloudflare describes it as “an authentication method that leverages
cryptographic signatures in HTTP messages to verify that a request comes from an automated bot”.
Every signed request carries three headers: Signature, Signature-Input and Signature-Agent,
the last pointing to your key directory. The directory serves a JSON Web Key Set over HTTPS and is
itself signed. The key pair is Ed25519, made with OpenSSL as Cloudflare’s guide shows:
openssl genpkey -algorithm ed25519 -out private-key.pemopenssl pkey -in private-key.pem -pubout -out public-key.pemThe guide then converts the public key to JWK format, hosts the directory, and registers the bot in the dashboard’s Bot Submission Form with “Request Signature” as the verification method. The work builds on two IETF drafts, one for the key directory and one for the protocol.
Categories and signed agents
Cloudflare sorts verified bots into categories that site owners can allow or block: Search, Agent, Training, Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring & Operations. Older category names keep working in WAF rules.
Agents acting for people (a browsing assistant, say) can be verified too. “As of July 1, 2026, the distinction between a Verified bot and a signed agent is expressed by a new metadata field tracked in BotBase”, which records whether one operator uses it directly or it serves many end users. Approved bots appear in BotBase and in Cloudflare Radar’s bots directory.
What this means if you run a scraper
- Most scrapers don’t qualify, and that’s fine. Verification is for services that identify themselves to every site and follow each site’s rules. A pipeline that collects data from a few sites you have permission to automate is better served by asking those sites to allow it.
- If you do qualify, apply. A monitoring service, a feed reader or a search crawler with its own addresses and a unique user agent can be verified, and then sites can let it through by category instead of challenging it.
- Never borrow an identity. Sending Googlebot’s user agent from your own addresses is exactly what verification catches: the claim and the address don’t match.
- Respect robots.txt and crawl-delay either way. It is a requirement for verified bots and good practice for everyone else. See responsible captcha automation.
ZeroCaptcha is not a verified bot and does not make your client one. It solves Cloudflare Turnstile widgets on the Cloudflare Turnstile solver and Cloudflare challenge pages on the Cloudflare WAF and 5-second challenge solver, for sites you are allowed to automate. How Bot Management scores the rest of the traffic is in Cloudflare Bot Management vs Cloudflare Turnstile.
Sources
- Cloudflare bots: verified bots (checked 1 October 2026).
- Cloudflare bots: Web Bot Auth (checked 1 October 2026).
- Cloudflare bots: IP validation (checked 1 October 2026).
- Cloudflare challenges: JavaScript detections, for the example rule (checked 1 October 2026).
- IETF drafts: HTTP message signatures directory and Web Bot Auth architecture (checked 1 October 2026).
The team that builds and runs the ZeroCaptcha API. Articles are drafted with AI tools, then checked against the API's code and the primary sources each one cites.