Short answer: Often, yes. If a client site sits behind Cloudflare, a bot-blocking setting (Bot Fight Mode, Super Bot Fight Mode, a managed challenge, or a custom WAF rule) can return a 403 or a challenge page to OpenAI's crawlers before they see a single page. The fix is to allow OpenAI's bots in Cloudflare, check that robots.txt does not also block them, and then re-test.
Why does ChatGPT fail to read a site that loads fine in my browser?
You are a human with a normal browser. OpenAI's crawlers are bots, and Cloudflare decides how to treat bots before the request reaches WordPress. According to OpenAI's crawler overview, it runs three user agents: GPTBot (training), OAI-SearchBot (search results) and ChatGPT-User (a visit triggered by a person asking ChatGPT about a page). A block on any of them can produce the "I can't access that page" reply.
Two separate layers can stop them, and agencies often fix only one:
- Network layer: Cloudflare returns a 403, a 429 or a JavaScript challenge. The bot never reads the page.
- Policy layer: robots.txt, or a meta tag, tells the bot to stay away. A well-behaved bot reads the rule and obeys it.
Which Cloudflare settings block ChatGPT?
| Setting | What it can do to AI bots | Where to look |
|---|---|---|
| Bot Fight Mode / Super Bot Fight Mode | Challenges or blocks traffic it scores as automated | Security, Bots |
| Custom WAF rule | A rule on user agent, country or ASN can block the crawler outright | Security, WAF, Custom rules |
| Managed robots.txt / AI bot blocking | Cloudflare can add disallow lines for AI crawlers or block them for you | Security, Bots |
| Rate limiting | Returns 429 when a crawler fetches many pages quickly | Security, WAF, Rate limiting rules |
Cloudflare's verified bots documentation describes a list of known bots that are checked and treated differently from random scrapers. Whether a given AI crawler is allowed depends on your plan and settings, so read the current page rather than assuming.
How do I test it without waiting for a client complaint?
Send a request that identifies itself as OpenAI's crawler and look at the status line and headers:
curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://client-site.example/A HTTP/2 200 is a good sign. A HTTP/2 403 with a header like cf-mitigated: challenge means Cloudflare is stepping in. One honest caveat: Cloudflare verifies real bots by more than the user agent, so a spoofed user agent from your own machine can be challenged even when the real crawler would pass. Treat a 403 as a strong warning and a 200 as a partial pass. For a quicker check across many bots at once, the free AI crawler checker reads robots.txt and the live response for you.
How do I fix it on a client's Cloudflare account?
- Open the zone in the Cloudflare dashboard, go to Security, then Events, and filter by user agent
GPTBot,OAI-SearchBotorChatGPT-Userto see which rule fired. - If Bot Fight Mode is the cause, decide with the client whether to turn it off or move to a plan where verified bots can be allowed.
- If a custom rule is the cause, edit it so OpenAI's user agents are skipped, or add an allow rule above it. See Cloudflare custom rules.
- If a managed robots.txt adds AI-bot disallow lines, switch it off or edit the policy for the bots the client wants to allow.
- Check robots.txt next (see below), then re-run the curl test.
Is robots.txt also blocking it on WordPress?
Open /robots.txt on the live domain and look for User-agent: GPTBot or User-agent: * followed by Disallow: /. SEO plugins, security plugins and staging leftovers all write these rules. Also open Settings, then Reading in WordPress. The "Discourage search engines from indexing this site" box is easy to leave ticked after a launch, and WordPress documentation describes it as a request to search engines, not a guarantee.
Remember what robots.txt does. Google's robots.txt introduction says it is mainly for managing crawler traffic, not a way to keep a page out of Google. OpenAI's overview lists its bots separately, so a client who says "we blocked AI" may have blocked more or less than they meant.
Can the same block hide a client site from Googlebot too?
It can, and that is the bigger risk. A broad challenge rule can catch any bot it does not recognise. Google's Googlebot documentation explains how to verify that a visitor really is Googlebot, so base allow rules on that, not on a user agent string alone.
While you are in the client's setup, check the other quiet blockers. A noindex meta tag or an X-Robots-Tag HTTP header can keep pages out of results; Google's block indexing guide describes both. Header check:
curl -sI https://client-site.example/ | grep -i x-robots-tagAny output containing noindex deserves a look. Then confirm the sitemap still lists the right URLs (see Google's sitemap overview) and run the page through the URL Inspection tool in Search Console, which Google's help page describes as showing how Google sees a URL.
What do I tell the client?
Keep it short: "Your Cloudflare protection treats ChatGPT's crawler as a threat, so it cannot read your pages. We can allow OpenAI's bots while keeping protection for everything else. Do you want your site readable by AI search, yes or no?" Some clients, such as publishers, really do want a block. That is their call. Your job is to make sure it is a choice, not an accident.
How does quietwatch catch this every day?
Cloudflare settings and robots.txt files change quietly, and nobody tells you when a crawler starts getting a 403. What quietwatch checks every day includes whether the major AI crawlers can still reach each client site, so you hear about it the day it changes. The free AI crawler checker above is the place to start.
Sources
- OpenAI: Overview of OpenAI crawlers
- Cloudflare docs: Verified bots
- Cloudflare docs: WAF custom rules
- Google Search Central: Introduction to robots.txt
- Google Search Central: Googlebot
- Google Search Central: Block search indexing with noindex
- Google Search Central: Sitemaps overview
- Search Console Help: URL Inspection tool
- WordPress documentation: Settings Reading screen