How do I check if ChatGPT can read my website?
Use HubSEO's free AI Crawler Access Checker to inspect your robots.txt and HTTP headers in real time. It determines whether GPTBot and OAI-SearchBot can access your content, verifies user-agent allow rules, and highlights any WAF or firewall rules blocking ChatGPT from indexing your pages.
What each agent actually controls, per its operator's documentation
When someone asks an AI assistant a question, retrieval agents fetch or look up public web pages to build answers with citations. Whether your site participates is controlled by each provider's documented agents, and blocking one does not block the others.
Search and retrieval agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot) affect whether your pages can be surfaced in AI answers. Training crawlers (GPTBot, ClaudeBot, CCBot) only govern model-training data. Data-usage tokens (Google-Extended, Applebot-Extended) do not crawl at all: they decide whether already-crawled content may train models or feed AI summaries.
| Agent | Operator | What blocking it does, per operator docs |
|---|---|---|
| OAI-SearchBot | OpenAI | Removes your pages from ChatGPT search answers (navigational links may remain) |
| Claude-SearchBot | Anthropic | Prevents indexing for Claude search, which may reduce visibility and accuracy in results |
| PerplexityBot | Perplexity | Removes your site from Perplexity search results |
| GPTBot | OpenAI | Opts your content out of OpenAI model training; does not affect ChatGPT Search |
| ClaudeBot | Anthropic | Opts future content out of Anthropic model training; does not affect Claude search |
| Google-Extended | Opts content out of Gemini training and AI Overviews use; does not affect Google Search |
Frequently Asked Questions
How does the AI Crawler Access Checker work?
The tool fetches your live robots.txt directly and evaluates exact User-agent match and wildcard fallback rules for each documented search, training and data-usage agent in HubSEO's crawler registry.
Can this check be run with JavaScript disabled?
Yes. All audit results are computed and server-rendered directly into the initial HTML response.
What should I do if a bot is marked as Blocked?
Review your robots.txt to remove unintentional Disallow directives or add explicit Allow rules for the affected user-agent.
Does this tool store my website data?
No. Checks are stateless diagnostic queries evaluated in real time without persisting user credentials or private data.