Short answer: Google found the URL through links, could not crawl it because robots.txt disallows it, and indexed it anyway, usually with no description. It is a warning, not an error. If the page belongs in Google, remove the robots.txt rule so Googlebot can read it. If it does not belong in Google, remove the robots.txt rule and add a noindex, because Google can only see a noindex on a page it is allowed to crawl.
What does "Indexed, though blocked by robots.txt" mean?
Search Console lists this reason in the Page indexing report. Google's help page describes it as a page that "was indexed despite being blocked by your website's robots.txt file". That sounds contradictory until you remember what robots.txt does: it stops crawling, not indexing. Google's own robots.txt introduction says it "is not a mechanism for keeping a web page out of Google". When other pages link to a blocked URL, Google can list the URL without ever fetching it, so the result shows the address with no snippet.
Did you mean to block it?
| What you want | What is happening | What to do |
|---|---|---|
| The page should rank | Googlebot cannot read the page, so it ranks on the URL alone | Remove the Disallow rule, then request indexing |
| The page should stay out of Google | The block hides the page's own noindex, if it has one | Remove the Disallow rule, add noindex, let Google recrawl, then block again if you must |
The whole site is blocked (Disallow: /) | A staging robots.txt usually went live with the site | Replace robots.txt with the production version today |
Where did the rule come from?
- A staging robots.txt pushed to production. The most common cause after a launch or a migration. The file has
Disallow: /for every user agent, so every page on the site shows this reason or drops out. - An old WordPress install. Before WordPress 5.3 the "Discourage search engines from indexing this site" setting wrote a Disallow into robots.txt. Since 5.3 it writes a
noindex,nofollowmeta tag instead, which is why Google changed its advice. A site that was set up years ago can still carry the old rule in a hand-written file. - A rule that is wider than intended.
Disallow: /blogalso blocks/blog-post-about-pricing, because robots.txt rules match the start of the path. - A plugin or a CDN rule. Some security plugins and hosting panels add rules of their own. Open the live file at
https://your-site/robots.txtand read it, rather than the copy in the CMS.
How do I check which rule blocks a page?
The free noindex checker reads robots.txt first and quotes the Disallow that keeps Googlebot off the page, next to the page's own noindex answers, so you can see both halves of this message at once. In Search Console, paste the URL into URL Inspection and look at "Crawl allowed?". For the file itself, open Settings and then the robots.txt report. It shows the robots.txt files Google fetched for the property, when it last fetched each one, and any errors or warnings in the rules. Google retired the old robots.txt tester in late 2023, so the report is now the place to look.
How do I fix it?
- Edit robots.txt on the server, not only in the CMS. On WordPress with Yoast or Rank Math the file can be edited in the plugin's tools page; on other stacks it is a file in the web root. Keep
User-agent: *with the rules you really want, and keep theSitemap:line. - Fetch the live file in a browser or with
curl https://your-site/robots.txtand confirm the rule is gone. A CDN may cache the old version for a while. - Ask Google to recrawl. For one page, URL Inspection and then "Request indexing". For the whole reason, open it in the Page indexing report and press "Validate fix". Google says validation typically takes up to about two weeks.
- If the page should stay out of Google, add
<meta name="robots" content="noindex">or anX-Robots-Tag: noindexheader, and leave the page crawlable until Google has seen it. Google's documentation is blunt about the order: a page that is blocked from crawling never has its noindex read.
How long does recovery take?
Once Googlebot can read the page again, a single page usually returns within days of the recrawl. A site that had Disallow: / for weeks takes longer, because Google has to crawl everything again and rankings settle gradually. The cost is in the weeks nobody noticed, which is the part worth fixing for good.
How do I stop this happening again?
Read the live robots.txt after every deploy, and have something read it for you on the days you do not. quietwatch fetches robots.txt for each site you add every day, works out whether Googlebot, Bingbot and the AI search crawlers are allowed on the home page, and emails you when the answer changes. You can check any site for free right now.