AI Crawlers and robots.txt

Six AI-crawler signals in the AI readiness check read your live robots.txt:

• GPTBot · OpenAI's training crawler
• OAI-SearchBot · OpenAI's search crawler
• ClaudeBot · Anthropic's crawler
• PerplexityBot · Perplexity's crawler
• Google-Extended · Google's control for Gemini training data
• Applebot-Extended · Apple's control for AI training

What permission means, precisely

Allowing a crawler in robots.txt means you permit it to fetch. That is all it means. Permission does not prove a crawler visited, does not guarantee inclusion in any answer engine, and does not control Google Search itself.

Equally, a request that spoofs a crawler's user-agent does not prove the real crawler was blocked or allowed. Identity claims without validated IP or DNS reverse-lookup are exactly that: claims.

Allowing the crawlers you want

To permit all six, either omit them from robots.txt entirely (allow-by- default) or list them explicitly:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

To keep a specific crawler out, use Disallow: / for that agent. Be deliberate: blocking answer-engine crawlers is a legitimate choice, but it does reduce the AI-readiness signals HubSEO can measure as passing.

The Google-Extended nuance

Google-Extended controls whether Gemini can use your site for model training. It does not control Google Search indexing, and it does not control AI Overviews. If you want pages in Google Search, manage that with meta robots directives and noindex, not with Google-Extended.

Verify, don't assume

After editing robots.txt, re-run the audit. The six ai-bot-* signals re-read the live file and the readiness score recomputes signal by signal. The deeper guide on the main site, guides/gptbot-robots-txt, covers per-agent strategy in more depth.