Short answer: Sometimes. As far as we can tell from Cloudflare's documentation, Bot Fight Mode is meant to leave verified bots such as Googlebot alone (see "Cloudflare Docs: Bot Fight Mode" in the Sources). Even so, real Googlebot traffic can be challenged or blocked when other settings stack on top of it: a custom WAF rule, a country block, a managed challenge on a path, a hosting firewall. Do not assume it is fine and do not assume it is broken. Test with Google's own tools, then loosen the specific rule that is catching Googlebot.
Why would Cloudflare block Googlebot at all?
Cloudflare's documentation describes Bot Fight Mode as a free setting that challenges traffic matching patterns of known bots. Its verified bots page (see Sources) lists Googlebot as an example of a bot it treats differently. Check both pages for the current wording, because plans and behaviour change.
In practice, agency sites collect layers over the years: a rate limit someone added after a scraping scare, a rule that challenges /wp-login.php and /wp-admin/ but was written too broadly, a country block, or a security plugin that blocks by user agent. Any of these can return a 403 or a challenge page to Googlebot. Google's page on HTTP status codes and network errors (see Sources) describes how it handles failed fetches. As we read it, errors that persist can reduce crawling and, over time, can lead to URLs dropping out of the index. Nothing looks wrong when you open the site in a browser.
How do I check whether Googlebot is being blocked?
Start with what Google sees, not what you see.
- In Search Console, open the URL Inspection tool and run a Live Test on the home page and one deep page. If the page fetch fails or shows a 403, a challenge page or an empty screenshot, you have your answer.
- Open the Crawl Stats report (Settings, then Crawl stats). A rise in 403 or 5xx responses, or a fall in total requests, around the date someone changed Cloudflare settings is a strong signal.
- In Cloudflare, open Security, then Events. Filter by action (challenge, block) and look for requests whose user agent contains
Googlebot.
You can also try a quick command from your own machine:
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/A 403 or a challenge page means the site challenges that request. Read this result carefully. Your machine is not a Google IP, so Cloudflare is right to be suspicious of a fake Googlebot. It tells you the site challenges unverified bots, not that real Googlebot is blocked.
If you would rather not run these by hand, the Cloudflare Googlebot check looks at how the site answers Google's crawler and flags the common Cloudflare causes.
Can I reproduce a real check of a suspicious request?
Yes. Google's guide to verifying Googlebot (see Sources) describes a reverse DNS lookup followed by a forward lookup, and it also points to a published list of Googlebot IP ranges. Take an IP from a blocked event in Cloudflare Security Events and run:
host 66.249.66.1
host crawl-66-249-66-1.googlebot.comThe guide's own example uses the IP above, and as we read it the first command should return a hostname ending in googlebot.com and the second should return the same IP. The exact output can change, so run it yourself rather than trusting a copy. If your blocked IP resolves back to googlebot.com or google.com, Cloudflare blocked real Googlebot and you have a rule to fix. If it does not, the block was correct. This is a reproducible method, not a story about a particular site.
What is the fix, step by step?
- Find the rule that fired. In Security, then Events, click the blocked Googlebot request and note the rule or feature named (Bot Fight Mode, a custom rule, rate limiting, a managed rule).
- If it was a custom rule or rate limit, add an exception for verified bots. Cloudflare's rules expressions support a verified bot field (
cf.client.bot), and the verified bots page (see Sources) explains what counts. - If it was Bot Fight Mode itself, check Cloudflare's current documentation on what it can and cannot be exempted from on your plan. If you cannot create an exception, the realistic options are to turn the setting off for that site or move to a plan with finer controls.
- On WordPress, also check any security plugin or host-level firewall. They sit outside Cloudflare and can block Googlebot even when Cloudflare is clean.
- Re-run the URL Inspection live test, then use Request Indexing on the key pages. Watch Crawl Stats over the next week.
| Where you see it | What it usually means | First move |
|---|---|---|
| URL Inspection live test fails | Googlebot is being blocked or challenged right now | Find the matching event in Cloudflare Security Events |
| Crawl Stats shows rising 403s | A rule change in the last days or weeks | Compare the date with Cloudflare's audit log |
| curl with Googlebot UA gets 403 | Unverified bots are challenged (expected) | Confirm with a Google tool, do not stop here |
| Cloudflare is clean but fetch fails | Host firewall or security plugin | Check the plugin and host logs |
What do I tell the client?
Keep it short and honest: "A security setting on your site was treating Google's crawler like a suspicious visitor. Visitors were never affected, which is why nobody noticed. We have adjusted the rule, confirmed Google can load your pages, and we are watching the crawl numbers for the next two weeks. Rankings can take time to recover once crawling is back." Do not promise a recovery date. As far as we know, Google does not publish one.
Where is the public evidence that this happens?
Site owners regularly post about it on public forums. The Cloudflare Community site has threads from people asking why Search Console reports pages Google cannot fetch after turning on bot protection. Search it for "Googlebot blocked" to read current examples. Each thread is one person's setup, not a rule that applies to every site. These are public posts by other people, not quietwatch customers, and we do not cite any of them as our own client work.
How does quietwatch catch this?
quietwatch fetches each client site every day the way a crawler would and alerts you when Googlebot-style requests start getting challenged, blocked or served errors, alongside noindex tags, robots.txt blocks and broken sitemaps; what quietwatch checks every day lists each one. The free Cloudflare Googlebot check above is the place to start.