Is GPTBot Blocked? Check and Fix It

Short answer: GPTBot is blocked if the site's robots.txt has a Disallow rule for the GPTBot user agent (or a blanket User-agent: * block), or if a firewall such as Cloudflare refuses requests that identify as GPTBot. Open /robots.txt, look for GPTBot, then request the homepage with a GPTBot user agent and check the status code. Blocking GPTBot is a choice some clients make on purpose, so confirm with them before you change anything on their site.

What does GPTBot actually do?

GPTBot is OpenAI's crawler. OpenAI's crawler documentation describes it as the bot for collecting content that may be used to train its models, and lists two other bots that are controlled separately in robots.txt:

User agent	What OpenAI says it is for
`GPTBot`	Collecting content for model training
`OAI-SearchBot`	Surfacing sites in ChatGPT search results
`ChatGPT-User`	Fetching a page when a person asks ChatGPT to open it

This matters for the client conversation. Per that documentation, blocking GPTBot should not remove the site from ChatGPT search, because that is a different bot. Blocking OAI-SearchBot is the one that affects it. Agencies often block one while meaning to block the other.

How do I check robots.txt for a GPTBot block?

Fetch the file and read the rules that apply to GPTBot. Under the robots.txt standard (RFC 9309), a crawler follows the group that matches its name, and falls back to User-agent: * only if there is no GPTBot group.

curl -s https://client-site.example/robots.txt | grep -i -B1 -A3 "gptbot"

If the site blocks it, you will see something like this:

User-agent: GPTBot
Disallow: /

Disallow: / blocks the whole site. A line such as Disallow: /private/ blocks only that folder. If grep prints nothing, there is no GPTBot group, so the wildcard group applies. Check it for Disallow: / too, because that blocks every crawler without a group of its own. While you are in the file, look for a Sitemap: line, since a missing or stale sitemap reference is a common leftover from a rebuild.

Could Cloudflare or a firewall be blocking it instead?

Yes, and robots.txt will not show it. Cloudflare's AI bot controls can, according to its documentation and announcement post, block AI crawlers at the edge and can also add AI bot rules to robots.txt for you. Test the real response by sending a request with the GPTBot user agent string:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://client-site.example/

A 200 means the request got through. A 403, a challenge page or a 429 means something in front of the site is refusing it. A spoofed user agent from your own machine can be treated differently from the real GPTBot, so a 403 is a strong hint, not proof. Confirm in Cloudflare's security events.

If you would rather not run this by hand, the AI crawler checker reads robots.txt and tests the common AI user agents for you.

Does a GPTBot block affect Googlebot or Search Console?

Not by itself. Googlebot is a separate user agent, and Google's crawler overview lists its crawlers and tokens separately from OpenAI's. But the same mistakes that block GPTBot often hit Google too: a blanket Disallow: /, a noindex meta tag, or an X-Robots-Tag: noindex response header added by a server or CDN. Google's robots meta tag documentation describes noindex in both forms. Google also says in its robots.txt introduction that robots.txt is mainly for managing crawl traffic and is not the way to keep a page out of Google, which is why a noindex needs checking separately. To confirm Google's side, open Search Console, read the Page indexing report for "Excluded by 'noindex' tag" or "Blocked by robots.txt", and check that the sitemap you found above is submitted and read without errors.

How do I unblock GPTBot on a WordPress site?

Work through these in order, because several things can write to robots.txt on a WordPress site.

Check Settings, then Reading, for the "Discourage search engines from indexing this site" box. It is a different control, and WordPress describes it as a request to search engines, but if it is ticked the site has a bigger problem than GPTBot.
Look for a physical robots.txt file in the site root. If one exists, WordPress serves it instead of its own virtual file.
Check your SEO plugin's robots.txt editor, and any security or "AI bot blocker" plugin. These can add a GPTBot rule without anyone noticing.
Remove the Disallow line for the bot you want to allow, or change it to an empty Disallow:.
If Cloudflare is in front, turn off the AI bot blocking you do not want and clear any managed robots.txt rule that adds the block.
Reload /robots.txt in a private window, since caching plugins and CDNs can serve an old copy.

What should I tell the client?

Do not unblock first and explain later. Ask one plain question: do you want your content used to train AI models, and do you want to appear in ChatGPT search? Those are two separate answers. A typical outcome is to allow OAI-SearchBot and ChatGPT-User for visibility and keep GPTBot blocked for training. Write the decision down in the client file, because the next person to edit robots.txt will not know it was deliberate. Also say out loud that robots.txt is a request, not a lock: well-behaved crawlers follow it, but it does not hide content from anyone.

What does a deliberate GPTBot block look like?

This is a public example, not a quietwatch customer. The New York Times publishes a robots.txt that, when we last looked, listed GPTBot under a disallow rule. You can check the live file yourself, because publishers change these rules:

curl -s https://www.nytimes.com/robots.txt | grep -i -A2 "gptbot"

For a large publisher that is a policy decision about training use. On an agency client's site the same two lines are a problem only when nobody chose them, which usually means a plugin update or a CDN toggle added them.

How does quietwatch catch this daily?

quietwatch re-reads each client site's robots.txt and tests AI crawler access every day, so a new GPTBot block shows up the day it appears instead of when the client asks why nothing changed. See what quietwatch checks every day. The free checker above is the place to start.

Sources

All posts · The fourteen checks

Is GPTBot Blocked on Your Client's Site? How to Check and Fix It