Short answer: there are usually two separate problems. Google keeps its own copy of robots.txt, and according to its documentation that copy is generally refreshed within about 24 hours (Source 1). You can ask for a faster refresh through Search Console (Source 2). If you still see the old file in your own browser, a layer on your side is serving it: a CDN such as Cloudflare, a caching plugin, a physical file that overrides WordPress's virtual one, or a host-level cache. Find which one before you change anything else.
How do I tell whether the old file is Google's copy or my server's?
Fetch the file yourself, outside a browser, and read the headers. Browsers cache aggressively, so a browser view proves little. Here is a command you can run against any public site right now:
curl -sI https://example.com/robots.txt | grep -iE 'cache|age|last-modified|etag|server'
To see a real file, run curl -s https://www.google.com/robots.txt | head -n 3. At the time of writing the first line it returns is User-agent: *, which is Google's own public robots.txt. Swap in your own domain and compare the body with what you saved.
On a site behind Cloudflare, the headers can include cf-cache-status: HIT and an age: value. Cloudflare's documentation describes HIT as the response being served from its cache (Source 4). If you see that and the body is old, the CDN holds the stale copy. If the headers show MISS or DYNAMIC and the body is still old, the origin itself is the problem. If the body is correct, the old version you are worried about is probably Google's.
How long does Google keep an old robots.txt?
Google's documentation says its crawlers cache the file and that the cached copy is "generally" refreshed within 24 hours. It also says the cache may last longer when refreshing is not possible, for example because of server errors (Source 1). So a change made this morning may not apply to crawling until tomorrow. That is normal, not a fault.
To speed it up, Google's help page on submitting an updated robots.txt describes requesting a recrawl from the robots.txt report in Search Console (Source 2). Treat that as a request, not a guarantee.
Why does WordPress keep showing the old robots.txt?
WordPress serves a virtual robots.txt unless a real file exists in the site root. If a physical robots.txt is sitting in the web root (often left by an old SEO plugin, a migration or a developer), the server normally returns that file and WordPress never gets asked. Edits in your SEO plugin then seem to do nothing.
- Connect by SFTP or your host's file manager and look in the site root for
robots.txt. - Either edit that file, or rename it and let WordPress build the virtual one again.
- Check Settings, Reading. If "Discourage search engines from indexing this site" is ticked, WordPress adds a disallow rule to the virtual file and also marks pages noindex. It is easy to leave on after a staging launch.
- Clear your caching plugin and any host-level cache, then fetch the file again with curl.
Developers can also change the virtual output through the robots_txt filter (Source 5), so check the theme and mu-plugins if rules keep appearing that you did not write.
How do I purge a cached robots.txt on Cloudflare?
Cloudflare's documentation lists the file extensions it caches by default, and .txt appears on that list (Source 4). Purge the one URL rather than everything. Cloudflare's purge page describes a custom purge by URL in the dashboard (Source 3): choose the URL option and enter the full robots.txt address. Then run the curl command again and confirm the body is new. Also check whether Cloudflare's managed robots.txt feature is switched on, because it may add its own directives to what your origin serves.
Which layer holds the old copy?
| Where you look | What you see | Fix |
|---|---|---|
| curl against the live URL | Old body, cf-cache-status: HIT | Purge that URL in Cloudflare |
| curl, no CDN header, old body | Origin or plugin cache | Check for a physical file, clear plugin and host cache |
| curl shows the new body | Google may still act on its older copy | Request a recrawl in Search Console, allow up to about 24 hours |
What should I tell the client?
Keep it short and factual: "The new robots.txt is live on the server. Google says it generally refreshes its copy within a day, so crawling should follow the new rules soon. We have asked Google to recrawl the file." Do not promise an indexing date. Robots.txt controls crawling, and pages already indexed do not drop out or come back on a schedule you can set.
Before you report back, test the rules that matter. The AI crawler checker reads a live robots.txt and shows which crawlers are allowed or blocked, which is a quick way to confirm the file everyone else sees is the one you intended.
Here is a scenario to watch for. It is an illustration, not a quietwatch customer: a staging site is launched with Disallow: /, the developer fixes the file, but a CDN keeps serving the staging version. Nobody notices because the person who edited it sees the right file in their browser. A daily anonymous fetch, which is part of what quietwatch checks every day, shows the gap between the file you meant and the one crawlers get.
How does quietwatch catch this every day?
quietwatch fetches each client site's robots.txt daily from outside, the way a crawler would, and alerts you when it blocks Googlebot or other important crawlers, or when it changes. That catches a stale cached file as well as a bad edit. The free tool above is the place to start.