Short answer: As far as the public plugin documentation shows, W3 Total Cache does not write Disallow rules into robots.txt. When one appears, the cause is usually another plugin or the WordPress "Discourage search engines" setting, often copied over from a staging site. A physical robots.txt file on the server, or a cached copy of an old version, can also be to blame. Open the live file, work out which layer produced it, fix that layer, then purge every cache so Google fetches the corrected file.
How do I confirm what the live robots.txt says?
Do not trust the plugin screen. Fetch the file the way a crawler does:
curl -sI https://client-site.example/robots.txt curl -s https://client-site.example/robots.txt
Check the status code, the Content-Type and the body. The line that does the damage is usually this one:
User-agent: * Disallow: /
Under Google's robots.txt documentation, a Disallow: / rule for User-agent: * asks compliant crawlers not to crawl any part of the site (Google Search Central, "Create and submit a robots.txt file", listed in Sources). WordPress core is documented to produce this kind of virtual output when Settings, Reading, Discourage search engines from indexing this site is ticked (WordPress documentation, "Settings Reading screen", listed in Sources). Staging sites are often set up that way on purpose, and the setting can then travel with a database copy to production.
Which layer wrote the Disallow line?
WordPress builds robots.txt in layers, and each one can override the one before it. Go through them in this order.
| Layer | How to check | Fix |
|---|---|---|
Physical robots.txt in the web root | Look for the file over SFTP or in the host file manager | Edit it, or delete it so WordPress serves its virtual file |
| Reading setting ("Discourage search engines") | Settings, Reading in wp-admin, or the wp-cli check below | Untick it and save |
| SEO or other plugin with a robots editor | Check the robots.txt editor in each active SEO or security plugin | Remove the rule, or reset the file |
Code using the robots_txt filter | Search the theme and mu-plugins for robots_txt | Remove or correct the filter |
| Page cache or CDN | Compare the live file against an uncached request, such as one with a query string added | Purge the cache and exclude robots.txt from caching |
| Cloudflare managed robots.txt | Check the bots settings in the Cloudflare dashboard | Turn the managed content off, or confirm that it is intended |
If you want to see what a crawler can actually reach, run the site through our AI crawler checker. It reads the live robots.txt for named crawlers and shows which are blocked.
Can I reproduce the staging setting problem myself?
Yes. On any WordPress install you control, such as a local or throwaway test site, you can check the setting with wp-cli. This is the option behind the Reading screen checkbox:
wp option get blog_public 0
A value of 0 means "Discourage search engines" is on. Fetch /robots.txt from that site and you should see the Disallow: / rule shown above. Then flip it and fetch again:
wp option update blog_public 1 Success: Updated 'blog_public' option.
The virtual file should then fall back to WordPress's usual default, which the robots_txt filter reference documents as disallowing /wp-admin/ only. This is a test on your own site, not a report about anyone else's. If you copy a database from staging to production, run the same wp option get blog_public on production before you call the launch done.
Is W3 Total Cache ever the cause?
It can be, but indirectly. We have not found a robots.txt editor in the plugin's public documentation, so a Disallow rule written by the plugin itself would be unusual (see the plugin page in Sources). Two things are worth checking. First, a page cache or CDN setting can store an old robots.txt response and keep serving it after you have fixed the source. Second, a rule someone added by hand can end up in a physical robots.txt. Read the file before blaming the plugin. If the Disallow line is still there after you disable a plugin and purge caches, the plugin was not the source.
What are the exact fix steps?
- Save a copy of the current robots.txt so you can show the client what was there.
- Find the layer from the table above and correct it. Untick the Reading setting if it is on.
- Purge the page cache, the object cache, and any CDN or host-level cache. In W3 Total Cache this is the performance purge option. In Cloudflare, purge the robots.txt URL.
- Fetch the file again with
curland confirm the Disallow line is gone and the status is 200. - In Google Search Console, open the robots.txt report and request a recrawl of the file. Google's documentation says it "generally" caches robots.txt for up to 24 hours, so the fix can take a day to be picked up (Google Search Central, "Update the robots.txt cache", listed in Sources).
- Inspect two or three key URLs in Search Console to check that crawling is allowed again.
What should I tell the client?
Keep it short and factual. For example: "A setting left over from the staging copy told search engines not to crawl the site. We removed it on [date], cleared the caches and confirmed the live file is correct. Google rechecks the file about once a day, and pages that dropped out may be crawled again over the following days to weeks. We will check it again tomorrow." Do not promise a ranking recovery date. Google decides that, not you.
How do I stop it happening again?
Add a robots.txt check to your launch and migration checklist, and recheck after any plugin update or host change. Make sure the staging setting is reviewed every time a database is copied to production. It is also worth running a check on a schedule rather than relying on someone to remember.
How does quietwatch catch this?
quietwatch fetches each client site's robots.txt every day, along with its noindex tags, headers and bot settings, and emails you when a blocking rule appears. That is what quietwatch checks every day. The free AI crawler checker above is the place to start, since it shows you today's file in seconds.
Sources
- Google Search Central: Introduction to robots.txt
- Google Search Central: Create and submit a robots.txt file
- Google Search Central: Update the robots.txt cache
- WordPress documentation: Settings Reading screen
- WordPress developer reference: robots_txt filter
- Cloudflare docs: Managed robots.txt
- W3 Total Cache plugin page on WordPress.org
- WordPress developer resources: wp option (wp-cli)