Does a physical robots.txt override the WordPress virtual one?

Short answer: yes. If a real file named robots.txt exists in the WordPress root folder, the web server normally sends that file and WordPress never runs. The virtual robots.txt that WordPress builds on the fly, along with anything your SEO plugin or the "Discourage search engines" checkbox adds to it, is then ignored. Edits you make in the dashboard will appear to do nothing.

What is the WordPress virtual robots.txt?

A fresh WordPress install has no robots.txt file on disk. When a crawler requests /robots.txt, WordPress builds the response in PHP using its do_robots() function, which the WordPress developer reference describes as displaying the robots.txt content for the site (see Sources). Per that function's source, the output is a short default. On a fresh install, you can expect something like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://client-site.example/wp-sitemap.xml

The exact lines vary by WordPress version and settings, so treat the block above as a typical default, not a guarantee. If the site is set to discourage search engines, the source shows the output changing to Disallow: / instead.

Plugins such as Yoast SEO, SEOPress and Rank Math hook into this process, mostly through the robots_txt filter documented by WordPress, to add their own rules. That only works while the response is generated by WordPress.

Why does a physical file win?

WordPress is not asked first. On a typical Apache or Nginx setup, the server checks whether the requested path is a real file. If it is, the server returns it directly. Only when no file exists does the request fall through to index.php, where WordPress handles it. Server rules can differ, so confirm against your own host, but a real file normally stops every WordPress-side setting from reaching the crawler.

One related point from Google's documentation: it says the robots.txt file must be located at the root of the site host it applies to. So the root-level file is the one Google reads, whatever the dashboard shows.

How do you tell which one a site is serving?

Check the file on the server first. Look in the folder that holds wp-config.php over SFTP or your host's file manager. If you see robots.txt, it is physical.

You can also check from outside. This command prints only the response headers:

curl -sI https://client-site.example/robots.txt

A static file usually comes back with a Last-Modified header and a content type set by the server. A WordPress-generated response usually has no Last-Modified and a Content-Type of text/plain; charset=utf-8. Treat this as a hint, not proof, because caching layers and CDNs change headers. The file on disk is the real answer.

Then read the body with this command and compare it to what the dashboard says:

curl -s https://client-site.example/robots.txt

If your SEO plugin shows a Sitemap line and the live file does not, a physical file is probably winning.

An illustrative scenario (not a quietwatch customer): a site goes live from a staging copy, and the staging robots.txt, containing Disallow: /, came along as a real file. The Settings, Reading checkbox is unticked and everything in the dashboard looks fine, yet the command above prints Disallow: / on the live domain. Swap client-site.example for your own domain to run the same test.

How do you fix it?

1. Download a copy of the physical robots.txt as a backup.
2. Decide which version you want. For most client sites, that means letting WordPress and the SEO plugin manage it. Then delete the physical file.
3. If you must keep a real file (some hosts or security rules require it), edit that file directly and include the Sitemap: line yourself.
4. Clear page and server caches, and purge the CDN if there is one. Cloudflare and host-level caches can keep serving the old file.
5. Request /robots.txt again and confirm the content. Check that important URLs are not matched by a Disallow rule.
6. In Google Search Console, open the robots.txt report and confirm Google fetched the new version. Then use URL Inspection on a key page.

You can also run the site through our AI crawler checker to see which crawlers the live file allows or blocks.

One caution. Google's introduction to robots.txt says, in essence, that robots.txt is not a mechanism for keeping a page out of Google, and points to noindex or password protection for that. A robots.txt block stops crawling, but it does not remove pages already indexed, and a blocked page cannot show Google a noindex tag. Fix the file first, then let Google recrawl.

What do you tell the client?

Keep it short and factual: "A leftover robots.txt file on the server was overriding the settings in WordPress. We removed it and confirmed the live file now allows crawling. Google needs time to recrawl, so rankings may take days or weeks to recover. We will check the file again after every migration or host change." Do not promise a recovery date, because Google decides when to recrawl.

How does quietwatch catch this?

quietwatch fetches the live robots.txt from each client site every day, which is the file crawlers actually see, whatever WordPress or a plugin claims. You can see what quietwatch checks every day, and a change that blocks crawling raises an alert. The free AI crawler checker above is the place to start on a single site.

Sources

Claims attributed to Google or WordPress above come from these official pages. Documentation changes over time, so check the current wording.

• Google Search Central: Introduction to robots.txt (robots.txt is not a way to keep a page out of Google; noindex or password protection is)
• Google Search Central: Create and submit a robots.txt file (the file must sit at the root of the site host)
• WordPress Developer Resources: do_robots() (builds the virtual robots.txt output, including the default rules)
• WordPress Developer Resources: robots_txt filter (the hook plugins use to add rules)

All posts · The fourteen checks