Getting a bot verified by Cloudflare

On October 10, 2026, a test request from my crawler came back from Cloudflare with a new answer: this is a verified bot. It had taken about five months. You can see BixelBot’s listing on Cloudflare Radar.

BixelBot is the crawler behind Bixel, my side project. This is what the Verified Bots program is, what it took to get in as one person, and what I think it means for the web.

It matters because bots are a growing share of the web’s traffic. On Cloudflare’s network, people made about 70% of all requests through 2025 and bots about 30%. By October 2026 that was 61% and 39%.

What a verified bot is

Cloudflare sits in front of a large share of the web and decides, for each request, whether it looks like a person, a good bot or a bad one. A verified bot is one that Cloudflare has confirmed is open about who it is and what it does.

The policy has two parts.

• Identify yourself honestly. A site must be able to tell that a request really came from you, through a cryptographic signature, a published list of IP addresses with a stable user agent, or reverse DNS.
• Behave. Obey robots.txt, keep to reasonable rates, and do not work around what a site owner has asked for.

Being verified does not force anyone to let you in. Site owners can still block a verified bot, by name or by category. What it changes is the default. Without it, a small crawler is usually treated as an unknown and challenged.

Why I wanted it

BixelBot has a rule that it never works around a block. If a site serves a challenge, the crawler records that and leaves. It does not retry from another address and it does not solve CAPTCHAs.

That rule had a cost. Before approval, the crawler skipped every site behind a bot-protection vendor that had not admitted it. By September that was 27,288 companies behind Cloudflare alone. I could have reached many of them by pretending to be a browser. I did not want to build a record of honest evidence on a dishonest crawler, so the only option was to ask to be let in.

What it took

Most of the work was done before I applied.

• A public page. bixel.com/bot says who operates the bot, what it reads, what it will never read, how fast it goes and how to opt out.
• One name. A single user agent that links to that page.
• A dedicated address. The crawler first ran from my home connection. The policy requires addresses that are used only by the bot, so I moved it to a single cloud server and published its IP. The rule now is that an address is published before any traffic leaves it.
• Signed requests. Every request carries a signature under a newer standard called Web Bot Auth. The public key sits at a fixed address on bixel.com, and anyone can check a request against it.
• Manners in code. robots.txt is read before the first request, the rate is about one request a second per site, and an opt-out list is checked before every request.
• A stop switch. A health check runs every five minutes and stops the crawler on its own if it sees traffic going somewhere it should not.

The timeline

• April. The identity work shipped: the bot page, the opt-out and the robots.txt handling.
• Early May. I moved off my home connection to a dedicated server.
• Mid-May. I applied.
• Late May. I set up a daily check against Cloudflare’s test address that emails me when the answer changes.
• July to September. Fixes on my side, described below.
• October 8. Approved, and listed in Cloudflare’s bot directory.
• October 10. Live. The test address returned its verified answer, and a site that had been challenging the crawler stopped.

What I got wrong along the way

Nothing was rejected, but I found my own mistakes one at a time.

• A redirect. My site was redirecting the address of the public key to a version with a trailing slash. The standard requires the exact path. The key was there and nobody could fetch it properly.
• An unsigned side door. A secondary tool was sending requests without a signature. If you say every request is signed, every request has to be.
• A value that was too short. After approval the crawler was still not recognized. Checking my signatures against Cloudflare’s reference code showed a random value in them was 22 bytes where the code expected 64. I fixed it, and the crawler was recognized the next day. I cannot say for certain that this was the cause.
• Thirteen names. An audit found 13 scripts sending slightly different user agents. One identity now means one, and a test enforces it.
• My own skip list. After approval, the crawler’s list of protected sites still counted Cloudflare as unverified, and it re-skipped 441 sites before I noticed.

After all that, I did not turn the crawler loose on the sites it had been skipping. Being admitted made more crawling possible, and I did not take that as a reason to do more.

What this says about the web

The background to all of this is the chart at the top: automated traffic is growing quickly. That chart counts every kind of request. Other measures, which count differently, already put people in the minority.

• Cloudflare said in July 2026 that more than half of internet traffic is now non-human.
• In its 2025 review, people made up 47% of requests for web pages on its network, and bots that were not AI made up 44%.
• Imperva’s 2026 report put automated traffic above 53% across its own customers.
• Cloudflare also reported that 52% of crawler requests were for AI training by June 2026, up from 22% in spring 2025.

These come from different networks and count different things, so they cannot be combined into one number, but they point the same way.

The rules have moved with the traffic. In 2025 Cloudflare began asking new sites whether to allow AI crawlers and started testing a way for sites to charge for crawling. Since July 2026 it sorts AI traffic into search, agent and training, and a verified bot is allowed according to its category while unverified bots stay blocked by default. Since September 2026, new sites block training and agent traffic by default on pages that show ads.

Five things I take from having been through it:

The question is changing. For twenty years the web asked visitors to prove they were human. With this much automated traffic, and with agents acting for people, that question is less useful. The better ones are who you are, what you are here for, and whether you can prove it.

Identity is becoming something you can check. A user agent is a string anyone can type, and a signature cannot be faked that way. Web Bot Auth is built on a published standard and is being worked on in the open at the IETF, which matters, because it should not belong to one company.

Small operators can get in. I am one person. The requirements were about being honest and consistent, and none of them were about size.

Gatekeepers matter more. One company’s defaults now decide what a large part of the web looks like to anything that is not a person in a browser. I benefited from that system, and I still think it is worth watching closely.

Verified is only the minimum. It says the bot is who it claims to be and follows the rules. Whether it gives anything back to the sites it reads is a separate question, and it is the one I think about most for Bixel.

There is more on how the crawler fits into the rest of the project in Building a playground for AI-native learning.