Not slow. Not badly optimised. Unreadable by an identified, well behaved crawler that obeys robots.txt and says who it is. The bot protection responsible serves a holding page carrying noindex, nofollow, which is what an AI crawler sees when it visits.
Top organic results for “accountants in {city}” across 8 UK cities: Manchester, Birmingham, Leeds, Bristol, Nottingham, Liverpool, Glasgow and Edinburgh, London. Directories and ICAEW listing pages were excluded, since they are not firm websites. Each homepage was fetched once, along with its robots.txt and XML sitemap, and scored by the same engine that powers our free audit.
This is not a random sample of the profession. It is a sample of the firms a searcher actually finds, which is the population that matters for a study about visibility. Every percentage below is calculated against the 44 sites that could be read, never against 56, because reporting a site as failing a check we could never run would be dishonest.
Nothing tells a crawler what the page is about
Measured to first byte, including DNS, TLS and redirects
No JSON-LD anywhere on the homepage
Explicit Disallow for GPTBot, ClaudeBot and others
Google writes its own, usually worse
Discovery left entirely to internal links
Duplicates cannot be resolved
Five of the twelve unreadable sites returned a Cloudflare challenge instead of their homepage. That page is roughly 5KB, it says “Just a moment...”, and it carries a noindex, nofollow meta robots tag.
A browser solves the challenge in a second and nobody notices. GPTBot and ClaudeBot cannot solve a JavaScript challenge. They get the holding page, and the holding page tells them not to index it.
So a security setting, usually enabled by someone who never discussed it with whoever owns marketing, quietly decides whether a firm can appear in an AI answer. Nobody made that call. It came with the plan.
Separately, 4 of the 44 readable sites block AI crawlers deliberately, with an explicit Disallow: / for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. All 4 carry the same five agents in the same order, and not one of them blocks PerplexityBot. Four firms independently arriving at an identical list, with the same gap in it, is a template. Nobody sat down and chose this either.
The first pass reported that five firms carried noindex on their homepage. That would have been a serious accusation, and it was false. We were close to publishing it with the firms named.
The tell was that all five pages were within a few hundred bytes of each other. Five unrelated companies do not produce near identical homepages. They were all the same Cloudflare challenge page, and its noindex tag is correct behaviour: an interstitial genuinely should not be indexed. The firms had done nothing wrong. Our crawler had simply never reached their sites.
Checking that led to a real bug. Our robots.txt parser accepted whatever it was handed. Given an HTML challenge page at /robots.txt, it found no directives it recognised and returned its default, which is that every crawler is allowed. A confident all clear, derived from a file it never actually read.
That was live in our free audit and our crawler checker. It is fixed. The body is now checked to see whether it is a robots.txt at all, and when it is not, the answer is that we could not read it rather than a reassuring guess.
The crawler blocking findings survived the same scrutiny. Each was confirmed by fetching the file independently, and each turned out to be a genuine, explicit Disallow: /. Those numbers stand.
This section exists because the failure mode is the whole point. A tool that reports confidently on data it could not collect is worse than no tool, whether it errs towards alarm or towards reassurance. The only defence is checking a finding against the thing itself before publishing it, which is also why no firm is named anywhere on this page.
Look for GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If any carries Disallow: /, somebody decided you should not appear in AI answers. Confirm that was on purpose.
curl it, or use any tool that does not run JavaScript. If you get a challenge page, that is what an AI crawler sees, and it carries noindex.
41% of the sites we read had more than one or none. It is a two minute fix and it is the clearest signal you have about what a page is for.
Median here was 3.5 seconds. If yours is worse, that is crawl budget you are paying for and not spending on being indexed.
Top organic results for "accountants in {city}" across 8 UK cities: Manchester, Birmingham, Leeds, Bristol, Nottingham, Liverpool, Glasgow and Edinburgh, London. Directories, ICAEW listings and aggregator pages were excluded because they are not firm websites. This is not a random sample of the profession. It is a sample of the firms a searcher actually finds, which is the population that matters for a study about visibility.
Twelve sites could not be measured. Seven refused the request outright and five returned a bot protection challenge instead of their homepage. Every percentage in the findings is calculated against the 44 that could actually be read, never against 56, because reporting a site as failing a check we were never able to run would be dishonest.
The figure is time to first byte including DNS, TLS negotiation and any redirects, measured from a single location on one attempt. It is not a lab benchmark and a second run would move it. What it is good enough to say is that 26 of 44 sites took over three seconds to return their first byte, which is slow by any measure and is the kind of thing a crawler notices before a human does.
No, and we will not. The study reports distributions rather than a league table. Naming firms for technical faults is a cheap way to get attention and an expensive way to lose a profession's goodwill. Several flags also turned out to be measurement artefacts rather than real faults, which is the second reason.
Yes. The audit is free and ungated, the sampling frame is written above, and the raw counts are on this page. Run any of these domains through it yourself and you will get the same measurements, because they are measurements rather than opinions.