Short answer: find the line that sends the header and delete it. In nginx that is an add_header X-Robots-Tag line in a server or location block. In Apache it is a Header set X-Robots-Tag line in .htaccess or the vhost. Reload the server, confirm with curl -I that the header is gone, then ask Google to recrawl the affected pages.
How do you confirm the header is really being sent?
Do not trust the page source. An X-Robots-Tag is an HTTP response header, so it never shows up in the HTML and your SEO plugin can say "index" while the server says otherwise. Check the headers directly:
curl -sI https://client-site.example/ | grep -i x-robots-tag
If the site is affected you will see a line like this:
x-robots-tag: noindex, nofollow
No output means the header is not on that URL. Test a few URLs, not only the homepage, because rules are often scoped to a path, a file type or a hostname. Also test the www and non-www versions. Google documents the header and its accepted values on its robots meta tag and X-Robots-Tag page. If you want a quick check without a terminal, the free noindex checker reads the same header and the meta tag for a URL.
Where does the noindex header come from?
The header is almost never typed in by hand on purpose. It usually arrives with a staging config, a copied server file or a plugin. Work through the likely sources in this order:
| Source | What to look for | Where |
|---|---|---|
| nginx | add_header X-Robots-Tag "noindex" | Files in /etc/nginx/sites-enabled/ and conf.d/ |
| Apache | Header set X-Robots-Tag "noindex" | .htaccess, vhost files, apache2.conf |
| Hosting panel | A "discourage indexing" or staging toggle | The host's dashboard |
| Cloudflare | A Transform Rule that sets a response header | Rules, then Transform Rules |
| WordPress | Plugin or theme sending headers from PHP | Plugins list, functions.php |
On the server, a recursive search finds most of them: grep -ri "x-robots-tag" /etc/nginx/ /var/www/. Run it as root on the agency's own box, and ask the host to run it on managed hosting.
How do you remove it in nginx?
- Find the
add_header X-Robots-Tagline, or anyincludethat pulls one in. - Delete it, or comment it out with
#. - Check the config with
nginx -t, then reload withnginx -s reload.
One nginx detail catches people: add_header directives are inherited from the level above only when the current level defines none of its own. So a header set at the server level can disappear inside a location block that adds a different header, and the reverse can hide a leftover noindex in one location only. The nginx headers module documentation describes this rule and the always parameter. A line ending in always sends the header on error pages too, so search for both forms.
How do you remove it in .htaccess or Apache config?
Open the .htaccess in the site root and look for a line such as Header set X-Robots-Tag "noindex, nofollow". Delete it, or if you cannot find the source and need it gone now, add this below it:
Header unset X-Robots-Tag
Treat unset as a patch, not a cure. The real fix is removing the line that sets it. Check parent folders too, because Apache reads .htaccess files from every directory above the requested file. The mod_headers documentation covers set, unset and always. Apache changes in .htaccess apply on the next request, so no restart is needed.
What if the server config is clean but the header is still there?
Check the layers in front of and behind the server. Cloudflare can add response headers through a Transform Rule, which you can review in its response header modification docs. On WordPress, the setting under Settings, then Reading, called "Discourage search engines from indexing this site" is documented in the WordPress Reading Settings guide. It is a common reason for noindex after a staging site goes live. Some security and SEO plugins can also send headers from PHP. Deactivate them one at a time on a test copy and re-run curl -I after each.
How do you get the pages back into Google?
Google only drops the noindex after it recrawls the page and sees the header gone. Two things matter here. First, make sure robots.txt is not blocking those URLs, because a blocked page cannot be recrawled and Google will never see the fix. Google explains this on its block indexing page. Second, open the URL Inspection tool in Search Console, test the live URL, and request indexing for the key pages (see Google's guide to asking Google to recrawl your URLs). For the rest of the site, watch the Page indexing report in Search Console for the "Excluded by 'noindex' tag" count to fall. That can take days or weeks depending on how often Google crawls the site.
What do you tell the client?
Keep it short and factual. For example: "A server setting told search engines not to list your pages. It was probably left over from the staging setup. We removed it on [date], confirmed the pages now allow indexing, and asked Google to recrawl the main ones. Recovery in search is usually gradual, so we will check the indexed page count again in a week." Do not promise a ranking date. Say what was wrong, what you changed, and when you will check again.
How does quietwatch catch this daily?
quietwatch reads the response headers, meta tags and robots.txt of each client site every day, so a noindex that arrives with a server change or plugin update is flagged the next day instead of weeks later. You can see what quietwatch checks every day. The free noindex checker linked above is the place to start if you want to test one URL now.
Sources
- Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
- Google Search Central: Block Search indexing with noindex
- Google Search Central: Ask Google to recrawl your URLs
- nginx: ngx_http_headers_module
- Apache HTTP Server: mod_headers
- Cloudflare Docs: Response header modification
- WordPress Documentation: Settings Reading screen