Blog · Cloudflare blocking Googlebot · 5 min read · Updated 2026-10-28

Why does my sitemap return 403 on Cloudflare, and is it blocking Googlebot?

Your sitemap returns 403 behind Cloudflare and Search Console says it couldn't fetch it. Here is how to find the rule and what to tell the client.

Short answer: a 403 on /sitemap.xml behind Cloudflare usually means a Cloudflare rule is blocking or challenging the request, not that WordPress is broken. Switching on an AI crawler block should not stop the real Googlebot, because Cloudflare's documentation describes Googlebot as a verified bot. A custom WAF rule, a country block, a rate limit or a challenge setting can still catch it. Find the rule in Security Events, add an exception for verified bots on the sitemap path, then resubmit the sitemap in Search Console.

Why does a sitemap return 403 behind Cloudflare?

A 403 means the server understood the request and refused it. On a Cloudflare site, that refusal often comes from Cloudflare before the request reaches WordPress. In July 2025 Cloudflare announced that it would block AI crawlers by default for new domains (see the public example below), and it now offers AI Crawl Control settings. Many agencies then turned on extra bot settings across every client zone in one go. Sitemaps are a common casualty because they are a single URL that bots hit often, so they trip rate limits and path rules first.

Google's sitemap documentation describes the sitemap as a file Google fetches to learn about your URLs, so Googlebot has to be able to reach it (Build and submit a sitemap). If Cloudflare answers with a 403 or a challenge page, Search Console can report the sitemap as "Couldn't fetch", and new pages may be found more slowly.

Do AI crawler blocks stop Googlebot?

They should not. Cloudflare's AI crawler controls are aimed at AI training and scraping bots, and Googlebot is on its verified bots list. The trouble comes from the settings around them. Here are the usual culprits:

Setting	How it blocks Googlebot
Custom WAF rule on a path or user agent	A rule written to stop scrapers also matches the sitemap or a broad user agent pattern
Country block	Googlebot mostly crawls from US addresses, so a rule blocking that country can catch it
Rate limiting rule	Googlebot fetching many URLs in a short window can get a 429 or 403
Under Attack mode or a managed challenge	Bots cannot solve the challenge page
Block rule for "known bots" or an old bot list	Matches more than the AI crawlers you meant to stop

Do not assume it is one of these. Look at the logs first.

How do I confirm which rule is blocking it?

Run curl -sI https://example.com/sitemap.xml from your own machine. A 200 here only tells you humans get through.
Run it again with a Googlebot user agent: curl -sI -A "Googlebot" https://example.com/sitemap.xml. A 403 here does not prove real Googlebot is blocked, because your IP is not Google's and Cloudflare may treat the request as a fake. If the response carries a cf-mitigated: challenge header, a Cloudflare rule is acting on bot traffic.
In the Cloudflare dashboard open Security, then Events. Filter by path /sitemap.xml and look at the Service and Rule columns. That names the exact rule.
In Search Console, open the Sitemaps report and run a live test with URL Inspection on the sitemap URL. This shows what the real Googlebot sees.

You can also run the Cloudflare Googlebot check on the domain to see whether robots.txt and the sitemap answer bot requests normally.

How do I fix it without opening the site to scrapers?

Edit the rule named in Security Events. Add a condition that skips it when the request is a verified bot (the cf.client.bot field in the WAF custom rule expression).
If you do not want to touch the existing rule, add a separate Skip rule above it for the sitemap paths: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and /robots.txt.
Lift country blocks and rate limits for verified bots on those paths.
Keep the AI crawler block for the bots you actually want to refuse. That setting is separate from Googlebot.
Clear any cached 403 for the sitemap URL, then resubmit the sitemap in Search Console.

According to the WordPress 5.5 announcement, core generates /wp-sitemap.xml by itself, and SEO plugins add their own. Test the URL your site actually advertises in robots.txt.

What does a real Googlebot check look like?

Google's page on verifying Googlebot describes a reverse DNS lookup followed by a forward lookup. Its own example uses one of Google's crawler addresses:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

That output is the one Google's documentation shows. A request that fails this check is not real Googlebot, which is why your own curl test with a Googlebot user agent proves little. Paste an IP from your Cloudflare Security Events into the same two commands to see whether the blocked request was genuinely Google.

For a public case of why these settings were switched on so widely, see Cloudflare's own July 2025 announcement, where the company says it will block AI crawlers by default for new domains. That is a public Cloudflare announcement, not a quietwatch customer story.

What should I tell the client?

Keep it short and factual. Something like: "A security setting on your Cloudflare account was refusing automated requests to your sitemap, which Google reads to find new pages. We added an exception for Google's verified crawler, resubmitted the sitemap, and will recheck it. Your existing rankings were not removed, but new and updated pages may have been discovered more slowly during that window." Do not promise a recovery date. Google decides when it recrawls.

How does quietwatch catch this daily?

It requests robots.txt and the sitemap the way a crawler does and flags a 403, a challenge page or a block, so you hear about it the day a rule changes. See what quietwatch checks every day for the full list. The free Cloudflare Googlebot check above is the place to start.

Sources

All posts · The fourteen checks

Free tool

Is Cloudflare blocking Google?: run it on any page, free

No account. The same measurement quietwatch runs on every client site, every day.

Is Cloudflare blocking Google?