Is Cloudflare blocking Googlebot on your client's site?

Short answer: Usually not, but it can happen. Cloudflare's documentation describes a category of "verified bots" that includes Googlebot and says they are handled separately from unknown automated traffic (see the Cloudflare docs on verified bots in Sources). Even so, a custom WAF rule, a rate limit, a country block or a strict bot setting can still return a challenge page or a 403 to Google. If Search Console shows crawl errors or pages dropping out after a Cloudflare change, check this first.

Why would Cloudflare block Googlebot at all?

The trouble usually starts with a rule that you or a previous developer added and that does not exempt verified bots. Cloudflare's WAF custom rules docs describe rules that can block, challenge or skip traffic based on fields such as country, ASN, user agent and URL path, so a rule written without a verified bot exception can catch Google too. The usual causes:

• A custom WAF rule that blocks or challenges by country, ASN, user agent or URL path, with no exception for verified bots.
• A rate limiting rule set low enough that a burst of crawling trips it. Google's HTTP status code guidance says it may slow its crawling when it sees 429 or 5xx responses, so a strict limit can quietly cut how much of the site gets crawled.
• Bot Fight Mode or Super Bot Fight Mode set to block or challenge, combined with other rules that override the verified bot allowance. Behaviour depends on your plan and settings, so check the dashboard rather than assuming.
• A challenge applied to /wp-json/, /feed/ or /wp-admin/admin-ajax.php, which can break themes and plugins that Google renders.
• An origin firewall or WordPress security plugin that blocks Google's IP ranges, so Cloudflare is not even the culprit.

How do you check whether Googlebot is being blocked?

Start with Google's own view, because that is the test that matters.

1. In Search Console, open the URL Inspection tool for the homepage and one deep page, then run Test live URL. A block shows as a fetch failure, a 403, a 5xx or a page that renders as a challenge screen.
2. Open Settings, then Crawl stats, and look at the response breakdown. A jump in 403, 429 or 5xx responses after a date lines up with a Cloudflare change.
3. In Cloudflare, open Security, then Events, and filter by the user agent containing Googlebot or by the action Block or Managed Challenge. If requests that look like real Googlebot appear there, you have found the rule.

Here is a reproducible example you can run from a terminal. It sends a Googlebot-style request and prints only the response headers:

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/

If the first line comes back as HTTP/2 403 or 503, and a cf-mitigated: challenge header is present, a Cloudflare rule is acting on Googlebot-style traffic. Treat that as a hint only. Cloudflare says verified bots are identified by more than the user agent string, so a spoofed request from your laptop can be blocked even when real Googlebot passes. Confirm with the live test in Search Console.

To check that a visitor in your logs really is Google, Google's guide on verifying Googlebot describes a reverse DNS lookup. For example, running host 66.249.66.1 on an address from Google's crawler range should return a name ending in .googlebot.com, and a forward lookup of that name should return the same IP. Google also publishes its crawler IP ranges as JSON, linked from the same guide.

For a fast first pass on any domain, use the Cloudflare Googlebot check.

What does a blocked Googlebot look like?

How do you fix it?

1. In Cloudflare, go to Security, then Events, and click the blocked Googlebot event. It should name the rule that fired.
2. Edit that rule. Add a condition so it does not apply to verified bots, using the verified bot field in the rule expression, or add a skip rule above it for verified bots.
3. If the cause is rate limiting, raise the threshold or exclude verified bots.
4. Remove challenges from /wp-json/, static asset paths and sitemap URLs such as /sitemap_index.xml.
5. Check any WordPress security plugin for IP blocklists or bot rules, and look at the host firewall too.
6. Re-run the live test in URL Inspection on the homepage and two deep pages, then click Request indexing on the ones that failed.

Do not solve it by allowlisting anything that claims to be Googlebot by user agent alone. Google's verification guide notes the user agent can be faked, so use the verified bot signal or Google's published IP ranges instead.

What should you tell the client?

Keep it short and factual: "Our security layer was challenging Google's crawler on some pages, so Google may not have been able to read them. We changed the rule so verified Google traffic gets through and asked Google to recrawl. Recovery can take days to weeks, not hours." Do not promise a ranking recovery date. Say what changed, when, and what you will watch next.

How does quietwatch catch this?

quietwatch tests each client site every day for bot access problems, along with noindex tags, robots.txt blocks, header directives, sitemaps and certificates, so a Cloudflare change that locks out Google shows up in your inbox instead of in a traffic report. See what quietwatch checks every day. The free tool above is the place to start for a single domain. The commands and settings in this post are general techniques, not findings from any quietwatch customer.

Sources

• Google Search Central: Verifying Googlebot and other Google crawlers (reverse DNS check, published IP ranges, user agent can be faked)
• Google Search Central: Overview of Google crawlers and fetchers (the Googlebot user agent string)
• Google Search Central: HTTP status codes, and network and DNS errors (429 and 5xx can slow crawling)
• Cloudflare docs: Verified bots (verified bots handled separately from unknown traffic)
• Cloudflare docs: WAF custom rules (block, challenge and skip actions, and the fields rules can match on)

All posts · The fourteen checks