AI agents and automation, SEO and GEO, ROI-focused websites, and custom software built around your business.
Manuel Technologies
HomeFree toolsAI crawlers
( Free tool / Grow )

Is your site invisible to AI answers?

We fetch your robots.txt and resolve the rules that apply to six named AI crawlers, wildcards included. It takes about a second, and a blocked crawler is usually a one line fix nobody knew about.

We fetch your robots.txt and resolve the rules that apply to six named AI crawlers, including wildcard rules. Nothing is stored.

( Questions )

What people ask about AI crawlers.

Why does this matter?

A blocked crawler cannot read your pages, so the engine behind it cannot cite you, whatever your content says. It is the first thing to check before spending anything on being visible in AI answers, and it is a one line fix when it is wrong.

Which crawler belongs to which engine?

GPTBot is ChatGPT and OpenAI training. ClaudeBot is Claude. PerplexityBot is Perplexity. Google-Extended controls whether your content grounds Gemini and AI Overviews. Applebot-Extended covers Apple Intelligence. CCBot is Common Crawl, which feeds many models indirectly.

I have no robots.txt. Is that a problem?

Not for access, because everything is allowed by default. An explicit file is still better: it removes ambiguity, lets you name crawlers deliberately, and gives you somewhere to declare your sitemap.

Should I block AI crawlers?

It is a real business decision, not an obvious yes or no. Blocking protects content from being used as training data. It also removes you from the answers those systems give, at a point where more buyers start there. Publishers often block. Service businesses selling expertise usually should not.

Does allowing them guarantee I get cited?

No, and anyone who tells you otherwise is selling something. Access is necessary and nowhere near sufficient. The page still has to be worth quoting, which is a content and structure problem rather than a robots.txt one.