AI agents and automation, SEO and GEO, ROI-focused websites, and custom software built around your business.
Manuel Technologies
HomeGlossarySearch and AI visibility
( Search and AI visibility )

What is AI crawler?

An AI crawler is a bot that reads websites on behalf of an AI system, either to train it or to fetch pages when it answers a question.

The main ones name themselves: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, Google-Extended for Google's AI products, and Applebot-Extended. They obey robots.txt, so a single Disallow line removes a site from every answer those systems give.

Many sites block them without a decision ever being made, because a security plugin or a hosting template shipped the rule. Our free crawler check reads your robots.txt and reports exactly which of these bots are allowed, which is the first thing to know before any other AI visibility work.

The check takes seconds and the fix is one line, which is why it comes first.