AI crawler checker: is GPTBot or ClaudeBot blocked on your site?
Paste a web address. We read its robots.txt the way a crawler does and show, for 14 AI crawlers, whether each one is allowed or blocked and by which line, plus whether a CDN wrote the rules. No account, no sign-up.
Requests are sent as QuietwatchBot, never as another bot. Free: 5 checks an hour per address.
Why check which AI crawlers can read your site?
Two kinds of bot visit a site on behalf of AI products. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the "-User" fetchers) read pages to answer questions and cite the source; block them and the site disappears from those answers. Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider and the opt-out tokens Google-Extended and Applebot-Extended) collect text for models; blocking them is a choice. The tool reads robots.txt the way each crawler does and shows the line that decides, so you can tell a deliberate opt-out from an accident.
| Crawler | What it does | If blocked |
|---|---|---|
| OAI-SearchBot, ChatGPT-User | ChatGPT search and live fetches | Not cited in ChatGPT answers |
| Claude-SearchBot, Claude-User | Claude search and live fetches | Not cited in Claude answers |
| PerplexityBot, Perplexity-User | Perplexity index and fetches | Not cited in Perplexity |
| GPTBot, ClaudeBot, CCBot, Bytespider | Model training | A choice; search is unaffected |
| Google-Extended, Applebot-Extended | Opt-out tokens for training | Googlebot and Applebot keep crawling |
What this tool cannot see
A CDN can block a crawler at the firewall with a 403 that no robots.txt line shows. Cloudflare's "Block AI bots" setting does both: it writes a managed section into robots.txt and it refuses the same crawlers at the edge. We only ever send requests as our own crawler, never with another bot's user agent, so the firewall part is invisible to us. If the site is behind Cloudflare, run the Cloudflare and Googlebot check as well.
How to use it
- Paste the address of the page you care about and press Check this site.
- Read the verdict at the top of the result, then the evidence under each answer: the exact line, tag, header or date.
- Follow the fix under any failing answer, run the check again, then keep the result link; it works for 30 days.
Questions
What is the difference between a search bot and a training bot?
A search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot and the -User fetchers) reads pages to answer questions and cite the source. A training bot (GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Applebot-Extended) collects text to train models. Blocking a training bot is a choice. Blocking a search bot hides the site from that assistant's answers.
Does robots.txt actually stop GPTBot or ClaudeBot?
OpenAI, Anthropic, Perplexity and Google say their crawlers obey robots.txt. The file is a request, not a lock: a crawler that ignores it can still fetch pages, and only a firewall rule stops that.
Why does the tool say a CDN is writing the rules?
Cloudflare's "Block AI bots" setting adds a managed section to robots.txt that names several AI crawlers. The same setting can refuse those crawlers at the firewall with a 403, which no robots.txt line shows and which cannot be tested from outside without pretending to be the bot. We never do that.
Can you test whether the firewall blocks a bot?
No. We only send requests as our own crawler, QuietwatchBot, never with another bot's user agent. For Googlebot, Search Console's live test is the honest way to see a firewall block; for AI bots, the vendors publish their IP ranges and your server logs show the 403s.
Which path does the check apply to?
The exact path you typed. A robots.txt rule can allow the home page and block /blog, so check the page you care about.
How often should I check this?
Every time robots.txt or a CDN setting changes, which is the problem: nobody tells you. quietwatch reads robots.txt for every site you add once a day and emails you when an AI crawler flips from allowed to blocked, or back.