Short answer: X-Robots-Tag is an HTTP response header that carries the same instructions as a robots meta tag, including noindex. It is set by the server, the hosting platform or a CDN, so it never appears in the page's HTML or in the CMS. If Google says a page is noindexed and you cannot find the tag, check the headers with curl -I.
What is the X-Robots-Tag header?
Google documents two ways to tell crawlers not to index a page. One is the robots meta tag inside the HTML. The other is the X-Robots-Tag header in the HTTP response, and any rule that works in the meta tag works in the header too. A response that carries it looks like this:
HTTP/1.1 200 OK
X-Robots-Tag: noindexThe header exists because some files have no HTML to put a tag in: PDFs, images and videos. That is its intended use. The trouble starts when it lands on HTML pages by accident, because nothing in the page or the CMS shows it.
Why can't I see it in WordPress or in the page source?
Because it is not in the page. The header is added when the response leaves the server, by one of these:
| Where it comes from | Typical case | How to confirm |
|---|---|---|
| Hosting staging environments | Kinsta documents that its staging and temporary URLs carry X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, and that the header cannot be removed on those URLs | The header disappears on the production domain; if it does not, the site is still being served as staging |
| Preview deployments | Vercel adds X-Robots-Tag: noindex to every preview deployment, and a previous production deployment starts returning it once replaced | Check the header on the production domain, not the preview URL |
| Server configuration | An Apache Header set X-Robots-Tag line in .htaccess or a config file, or an nginx add_header X-Robots-Tag, copied from staging during a migration | Search the config files for X-Robots-Tag |
| CDN or edge rules | A response-header rule on a CDN that was meant for one path and matches more | Compare the header with the CDN bypassed, or review the rules |
| Plugins | Yoast SEO sends noindex, follow only on its XML sitemaps, which is correct; a stray header on normal pages comes from somewhere else | Disable plugins one at a time only after the server and host are ruled out |
How do I check a page for it?
Quickest: the free noindex checker reads the headers of the page and of every redirect hop and quotes any X-Robots-Tag it finds; the X-Robots-Tag checker is the same engine with the header as its first question. By hand:
- From a terminal:
curl -I https://your-site/pageand read the response headers. On Windows PowerShell typecurl.exe, because plaincurlthere is an alias for a different command. - In Chrome: open DevTools, the Network tab, reload, click the first request (the document) and read "Response Headers".
- In Search Console: URL Inspection, "Test live URL", then "Indexing allowed?". When the answer is No, the result normally names where the noindex was found: in the robots meta tag or in the X-Robots-Tag header.
Two details worth knowing: HTTP header names are case-insensitive, so x-robots-tag counts; and Google's documentation says that when the meta tag and the header disagree, the more restrictive rule wins. A page with a clean meta tag and a noindex header is noindexed.
How do I remove it?
- Host-injected on staging: move the site to its production domain. On Kinsta the header is tied to the staging and temporary hostnames, so a custom domain is the fix.
- Apache: find and delete the
Header set X-Robots-Tagdirective, then restart or reload. Google's own example looks like this, scoped to PDFs; a copy without the scope hits every page:
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>- nginx: the same idea,
add_header X-Robots-Tag "noindex, nofollow";inside alocationblock. Remove it and reload. - CDN: edit or delete the response-header rule, then purge the cache.
- Then tell Google: run the live test again until "Indexing allowed?" says Yes, then "Request indexing". For many pages, "Validate fix" on the "Excluded by 'noindex' tag" reason in the Page indexing report.
Why does this one hurt more than a meta tag?
A meta tag shows up in the page source, in SEO plugin panels and in most site-audit tools. A header shows up only if someone looks at headers. Migrations are where it bites: the server config or the host's staging behaviour comes along with the site, the HTML looks fine, and the pages drop out of Google one by one as they are recrawled. "Excluded by 'noindex' tag" in Search Console is usually the first sign, often weeks later.
How do I catch it on day one?
Check the headers of the home page and a few key pages after every deploy, or let a monitor do it. quietwatch reads both the robots meta tag and the X-Robots-Tag header on every site you add, every day, and emails you the day a noindex appears. You can check a page for free right now.