What it checks
We evaluate your robots.txt for five user-agent tokens whose operators document them: GPTBot and OAI-SearchBot (OpenAI; the second is what ChatGPT search uses to find pages), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Google's token for AI training use). For each we record whether it is allowed, which group decided it (a named group or the wildcard) and the rule that matched. We also look for a section written by a CDN, such as Cloudflare's managed robots.txt, and report exactly which crawlers that section blocks, because nobody edited the file to get it.
Why it matters for search engines
Google-Extended controls whether your content is used for Gemini training and grounding; it does not change how Googlebot crawls or ranks pages, so blocking it has no direct search effect. Blocking the others has no search effect either. The check matters for search only in the indirect sense that a wildcard Disallow written for one purpose often blocks more than intended.
Why it matters for AI assistants
The operators of these crawlers state that they honour robots.txt. A block means your pages cannot be fetched for answers or citations by that assistant, whichever policy you meant. Both outcomes are legitimate; the failure is not knowing which one you have, especially when an agency looks after the file for a client.
How to fix it
Decide per crawler and write it down. To allow one, give it a named group with Allow: /. To refuse one, give it Disallow: /. Leave the wildcard group for everyone else. Then re-check: a named group overrides the wildcard for that crawler only.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Allow: /
User-agent: *
Allow: /Check one page right now, no account: the free AI crawler checker.
What quietwatch reports
ok when the blocks, if any, were written by hand: that is state, not a verdict, because blocking an AI crawler can be exactly what you intend. warn when a CDN's managed section is doing the blocking, naming the crawlers and quoting the lines, because that usually happened without a decision. unknown when robots.txt answers 401, 403 or 5xx to our own User-Agent. Daily monitoring alerts when the list changes.
Sources
- OpenAI, Overview of OpenAI crawlers
- Anthropic, Does Anthropic crawl data from the web?
- Perplexity, crawlers
- Google, overview of Google crawlers and fetchers (Google-Extended)
- Cloudflare, Managed robots.txt
All twelve checks · Previous: Bot access · Next: HTTP status