Robots.txt Checker
Your robots.txt in plain words. It is a small file that is easy to get wrong in ways nothing warns you about — a rule written before its group, an empty Disallow that means the opposite of what it looks like — and the cost of getting it wrong is pages quietly dropping out of search.
What this checks
- Whether the file exists, and whether what came back is really a robots file rather than your home page served for an unknown address.
- Every group of rules, which crawlers it applies to, and what each Allow and Disallow line actually permits.
- Syntax problems: a line with no colon, a rule written before any User-agent, an empty User-agent, an empty Sitemap.
- Sitemap references, and directives like Crawl-delay that are widely written but are not part of the standard.
What this cannot tell you
- This reads the file. It cannot tell you whether any crawler obeyed it — robots.txt is a request, and a badly behaved crawler simply ignores it.
- A rule here does not control indexing. A page you disallow can still appear in search results if other sites link to it, because the rule asks a crawler not to fetch the page rather than not to list it.
- Different search engines extend the standard in different ways. The matching here follows the rules the major engines document; a crawler with its own ideas may behave differently.
Questions people ask
Is robots.txt a way to hide private pages?
No, and treating it as one is a real risk. The file is public, anybody can read it, and listing a private path there tells the world exactly where to look. It is also not access control: a disallowed page is still reachable by anybody with the address. If a page must be private, it needs a password, not a line in this file.
Will disallowing a page remove it from Google?
Not reliably. The rule asks a crawler not to fetch the page, which is different from asking a search engine not to list it — a disallowed page linked from elsewhere can still appear, with no description because nobody was allowed to read it. To keep a page out of results, let it be fetched and use a noindex instruction on the page itself.
What does an empty "Disallow:" mean?
It means allow everything, which is the opposite of what most people reading it assume. "Disallow:" with nothing after it blocks nothing at all, while "Disallow: /" blocks the whole site. One character between the two.