Building a playground for AI-native learning

Bixel is the project I work on for fun. It keeps a dated, sourced record of what private companies are doing, read from the pages they publish themselves. It is also where I have learned the most about building with AI, because I had to build every part of it: the crawler, the bots, the extraction, the checks, the API.

This note is the long version of what is in it and what it has taught me. The bigger reason for the project is in Why I’m Building Bixel.

It started as something else

The first commit is from February 2026, and the first version had nothing to do with company data. It was a site for evaluating AI models on design tasks: 40 models, 400 prompts, a scoring model and head-to-head voting. I spent March adding leaderboards and comparison pages.

By April I was more interested in a different question, which was what you could learn about a company by reading its own website carefully. I ran a first crawl of 20,000 sites, and a week later I deleted the old product: 22 database tables and everything built on them. The repository is still named after the model evaluator.

Since then it has been more than 2,500 commits, and July alone was 619. The code is private, but the activity is on Bixel’s GitHub profile.

A crawler I would be comfortable receiving

The crawler is called BixelBot, and most of the work in it is restraint.

One identity. It uses a single user agent that links to a public page explaining what it does. Every request is cryptographically signed, so a site can prove the request came from Bixel.
One address. It ran on a Mac Mini at home until May, then moved to a single cloud server with one published IP address. I had planned to use proxies and dropped the idea the same week.
Manners. It checks robots.txt before the first request, keeps to about one request a second per site, and checks an opt-out list before every request.
It takes no for an answer. When a site serves a challenge or a block, the crawler records it and moves on. It never retries around a block and never solves a CAPTCHA.

That last rule cost me a lot of coverage for months, and it is the reason I applied to Cloudflare’s Verified Bots program. I wrote about that separately in Getting a bot verified by Cloudflare.

More than the crawler

Websites do say a lot. Bixel has read 10,886 software companies in full, and I counted what each one publishes on its own site. Nearly all describe what they are, and 78% show pricing tiers. Fewer talk about security: 40% say anything, 12% name a SOC 2 report, and 5% run a trust center, which is a page where a company posts its security and compliance documents. About a third state their legal name, and 4% give a registration number.

Technology is the part read from the page’s code and not from what the company says. Bixel has scanned 900 of these companies so far and found 490 different technologies across them.

Company websites are only one source. To say anything reliable about a company you also need to know which legal entity it is, and a website will rarely tell you that. So Bixel reads public records too.

Business registries. The identity layer is built from GLEIF, the global register of legal entity identifiers, and Companies House in the UK. In September I loaded 8.3 million registry entities from those two, and about 99,000 of them matched across both. I have since recorded the terms for four US state registers: California, Colorado, New York and Florida.
The SEC. Bixel reads EDGAR for registrant records and Form D filings. A funding round from a Form D sits beside the round the company announced itself, each labelled with where it came from.
Web archives. Common Crawl and the Wayback Machine fill in history from before Bixel was watching.
Lists and logs. The Tranco ranking seeds which sites to visit, Certificate Transparency logs reveal subdomains, and Wikidata helps identify public companies.

Two outside services do work I chose not to build.

Exa for search. I use Exa’s search API to find pages, and only for that. It names the address, BixelBot goes and captures the page, and every fact cites that capture and never Exa’s text. A search result is a lead, and I did not want leads stored as evidence.
OpenRouter for models. One adapter lets any extractor call a model from another provider, with data collection by providers switched off. That adapter is what made the model comparison below possible, and the cheaper model I moved to runs through it. It has costs: there is no prompt caching and no batch pricing. The first parallel test also died on a payment error, because I had not bought credits.

The lesson I did not expect was about rights. Every source has terms, and “public” does not mean “free to republish”. Bixel keeps a registry of each source’s terms in code, and a source whose terms I have not verified is blocked from the published dataset by a test. Two sites, LinkedIn and Crunchbase, are never fetched at all, and a test enforces that too.

Keep the evidence

The most expensive lesson came in June. I had a detector for which sites run on Shopify, and its pattern matched any page that mentioned the word. Of 8,032 detections, 5,443 were wrong. One of them was stripe.com.

The cleanup made it worse. I wrote a script to strip the false positives, tested it against one negative example, and ran it. It removed real detections along with the false ones, and I could not get them back, because at that point I was not storing the pages I had read.

Three rules came out of that week, and they now shape the whole system:

Every page the crawler reads is stored as it was received, under a key made from a hash of its content.
The stores are append-only. Nothing is edited in place.
Each batch of captures is chained to the one before it and timestamped, so the record can be re-checked later.

I am careful about what that proof claims. It shows that a page was observed at a certain time. It says nothing about whether my reading of the page was correct. That is a separate problem.

Reading pages with LLMs

A company record in Bixel has ten dimensions: pricing, features, security, reliability, hiring, stack, positioning, funding, location and legal. Each one has its own extractor, and they all work the same way.

The model is given the captured page and a form to fill in, at temperature zero.
Every value it returns has to come with the quote from the page that supports it.
“Unknown” is an allowed answer, and a welcome one.
The result is cached against the hash of the page, so an unchanged page is never read twice.

Each fact then carries where it came from, the date it was last supported by a capture, and a pointer to that capture. What a company states about itself is kept apart from what Bixel observes directly, such as what is detected on the live site or in DNS. The code that compares the two will fail before it will make up the observed side.

Changes get the same caution. When a value changes, Bixel holds it as pending until a second, newer capture confirms it. If the page reverts, the change is thrown away.

Testing models against each other

In October I ran a proper comparison of models for the extraction work. I hand-checked a set of pricing pages and scored nine models against them. These are five of the nine.

Model	Exact match	Tiers found	Cost per page
Sonnet 4.6 (what I was using)	64%	87%	about 0.5¢
Gemini 3.5 Flash-Lite	71%	73%	0.115¢
Haiku 5.5	57%	72%	0.02¢
qwen3-235b	57%	85%	0.057¢
gpt-4.1-mini	14%	90%	0.118¢

Here are all nine, with accuracy plotted against cost. The two to look at are the model I had been using and the one I moved to.

Exact match against cost per page

All nine models, on 14 hand-checked pricing pages. Higher is more accurate and further left is cheaper.

The sample is small, 14 pages, so I treat these as direction and not as a benchmark. What I learned from it:

The model I trusted was not perfect either. It matched exactly on 64% of pages. I had been treating it as the answer key.
The failures have shapes. Two models read a Chennai company’s rupee sign as Pakistani rupees. One multiplied a monthly price by twelve and reported a number that was not on the page. One wrote zero for tiers that had no price. Knowing the shape of a failure matters more than the score, because the shape is what you write a check for.
A tie is a win if it is four times cheaper. I moved seven dimensions to the cheaper model. I kept the more expensive one for features, where the cheap model stopped early on long lists.
Evaluation is cheap. The whole comparison cost 72 cents.

An earlier test in August had gone the other way. A cheaper model failed on all seven dimensions I tried, misreading uptime figures and inventing legal names. Models change quickly, and I now assume I will have to re-run the test.

What broke, and what I changed

Most of what I know about this system I learned from it being wrong.

Monitoring that lied. A critical “worker stuck” alarm fired while the crawler was capturing 8,771 pages an hour. It was measuring the wrong table. When I looked further, every health check defaulted to “healthy” when it timed out, so a failed check reported good news. I rewrote them so that no answer means not healthy.
A false alarm. On the day I moved the crawler to its new server, the logs showed a 0.3% success rate. The real number was 48%. The rest were sites the crawler had politely put back in the queue.
Fresh date, stale words. A written summary said a company had six open roles while the facts underneath said eight. Summaries now store a fingerprint of the facts they were written from, and are hidden when the two no longer match.
Prices that only exist after JavaScript runs. One page showed a $750 monthly price that was not in the HTML at all. I rendered 171 pages in a real browser to measure the gap. Plain HTML was missing 20.5% of pricing pages and 17.2% of hiring pages, and none of the features pages.
My own noise. When I audited the change feed, 380 of 1,209 confirmed changes were artifacts of my extractor and not anything the company had done. Separately, about half of all page changes turned out to be markup churn with identical text.
The wrong citation. The first time I tested Bixel through Claude, all 42 feature facts for one company cited its pricing page. Nine of them were only stated somewhere else. Nothing was invented, but a merge step had collapsed the sources, and for a product about provenance that is the same as being wrong.

Building it with agents

I built Bixel with AI coding agents, and the way I work with them has changed as much as the product.

A rules file where every rule has a history. Each rule is dated and names the incident that produced it. The Shopify cleanup is in there. So is a rule about checking spend before a large batch.
Rules that are enforced. The important ones are backed by tests. The crawler’s single identity is one of them, after I found 13 scripts sending slightly different user agents.
A fixed order for risky work. Dry run, look at the numbers, approve, then apply.
Numbers come from tools. Any figure written into a document has to come from the output of a command, not from memory.
A second model as auditor. I use one model to review the findings of another before I act on them.

What I take back to my day job

Three things have carried over into how I lead design.

The first is that the design work in an AI product is mostly about what happens when the model is wrong. The happy path takes an afternoon. Deciding what the system refuses to say, what it shows when it does not know, and how a person can check an answer is where the time goes.

The second is that you cannot judge this material from a distance. I had opinions about models before I ran my own comparison, and several of them were wrong.

The third is that evidence changes the conversation. When every value comes with a date and a source, people stop arguing about whether to trust the system and start using it.

If you want to look around, the docs are the best way in.