How do you block GPTBot but still allow OAI-SearchBot in robots.txt?

Short answer: give each bot its own group in robots.txt. Put Disallow: / under User-agent: GPTBot and Allow: / (or an empty Disallow:) under User-agent: OAI-SearchBot. OpenAI's documentation describes these as separate crawlers with separate controls, so blocking one does not block the other. The trap is a catch-all User-agent: * block above them, which can shut out the search bot too.

What is the difference between GPTBot and OAI-SearchBot?

As OpenAI's bots page describes them, GPTBot is used to collect content that may be used to train generative AI models, and OAI-SearchBot is used to surface sites in ChatGPT search results. A third, ChatGPT-User, fetches pages when a person asks ChatGPT to visit them. The page says robots.txt settings for each bot are independent. Check the live page before you promise a client anything, because these names and descriptions change.

What exactly goes in robots.txt?

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Under the robots exclusion standard (RFC 9309), a crawler follows the group that matches its name most specifically. A bot that has its own group ignores the * group, which is why the two named groups work even when the * group says something different. Google's robots.txt documentation describes the same grouping for its own crawlers.

Can you see a real site that blocks GPTBot?

Yes, and you can check it yourself. The New York Times was widely reported in August 2023 to have added a GPTBot disallow to its robots.txt. That is a public case, not a quietwatch customer. Publishers edit these lists, so run this to see the current file:

curl -s https://www.nytimes.com/robots.txt | grep -i -A2 gptbot

If GPTBot appears with a Disallow line beneath it, you are looking at the same pattern as the snippet above. If it does not, the publisher has changed its policy since.

Why does OAI-SearchBot still get blocked on some sites?

• A blanket block. User-agent: * followed by Disallow: /, and no named group for OAI-SearchBot. Staging leftovers are the usual source.
• A security or SEO plugin. Some plugins add AI-bot blocks in bulk and can list OAI-SearchBot next to GPTBot.
• A firewall that ignores robots.txt. A CDN or host rule that challenges unknown bots can stop OAI-SearchBot before it reads your file. The robots.txt looks right and the bot still fails.

How do you change it on a WordPress client site?

1. Open https://clientsite.com/robots.txt in a browser. Note whether a physical file exists in the web root or WordPress is serving a virtual one.
2. If it is virtual, WordPress generates it (see do_robots) and your SEO plugin usually has a robots.txt editor. Edit it there.
3. If a physical file exists, it wins. Edit the file over SFTP.
4. Paste in the two named groups above, keeping the existing * group.
5. Clear any page or CDN cache so the new file is served.
6. Check Settings, Reading and confirm "Discourage search engines from indexing this site" is off. It is separate from robots.txt, but a left-on box undoes the work.

What if the client is on Cloudflare?

Cloudflare's documentation describes a managed robots.txt feature that can prepend AI-crawler blocks to your file. If it is on, what is served may differ from what you edited, so fetch the live URL. Also review bot-fighting settings in the dashboard, since a firewall challenge can block a bot you allowed in robots.txt.

Does blocking GPTBot affect Googlebot, noindex or Search Console?

Not by itself, because Googlebot is a separate user agent. Three checks still belong on every client site. First, Google's documentation says robots.txt is not a way to keep a page out of Google and that a noindex rule is only seen if the page is not blocked from crawling (robots meta tag and X-Robots-Tag docs). Second, the same page describes X-Robots-Tag as the HTTP header form of noindex, so read response headers as well as the HTML. Third, the robots.txt report in Search Console shows which robots.txt files Google found and when it last fetched them. Also confirm the sitemap still lists the URLs you care about and is not disallowed.

How do you test that it worked?

curl -s https://example.com/robots.txt

Look for your two named groups and for any User-agent: * block with Disallow: /. Then request a page with the bot's name to catch server-level blocks:

curl -sI -A "OAI-SearchBot" https://example.com/

A 200 is good. A 403 or a challenge page means something in front of WordPress is refusing the bot. This only imitates the user-agent string, so it will not catch rules that verify the bot's IP range. For a quick read that checks the rules per bot, run the AI crawler checker.

What do you tell the client?

• "We have asked OpenAI's training crawler to stay away and left its search crawler allowed."
• "Robots.txt is a request that well-behaved bots follow. It is not a lock."
• "Allowing the search bot makes appearing in ChatGPT answers possible. It does not promise it."
• "A plugin update or a staging copy can undo this, so we recheck it."

How does quietwatch catch this every day?

A robots.txt that was right on Monday can be overwritten by a plugin update or a CDN setting by Friday. quietwatch checks each client site daily and tells you when a crawler that should be allowed becomes blocked; what quietwatch checks every day lists what that covers.

Sources

• OpenAI: Overview of OpenAI crawlers
• RFC 9309: Robots Exclusion Protocol
• Google Search Central: Create and submit a robots.txt file
• Google Search Central: Robots meta tag and X-Robots-Tag specifications
• Search Console Help: robots.txt report
• Google Search Central: Sitemaps overview
• WordPress Developer Resources: do_robots()
• Cloudflare docs: Managed robots.txt
• The New York Times robots.txt (public file used in the example)

All posts · The fourteen checks