AI Crawler Checker
Twelve AI crawlers, checked against your robots.txt. Most site owners who have blocked AI did it with one line that caught far more than they meant, and most who think they are open have never looked. This shows which rule applies to each crawler and which line in your file decided it.
What this checks
- Your robots.txt, fetched from the root of the site, and the group of rules that applies to each named crawler.
- The difference between crawlers that read pages so an assistant can quote and link you, and crawlers that collect pages for training. Those are separate decisions and most owners only meant to make one of them.
- Whether a crawler is blocked by a rule naming it, or caught by a catch-all that may have been written years before these crawlers existed.
- Permission tokens such as Google-Extended and Applebot-Extended, which are not crawlers at all and are shown separately for that reason.
What this cannot tell you
- It reads your robots.txt, which is a request rather than a lock. A crawler that ignores the file is not stopped by it, and this cannot tell you whether any particular crawler obeyed.
- Allowing a crawler does not mean an assistant will cite you, recommend you or know you exist. It means the door is open. What comes through it is not ours to promise.
- Blocking a training crawler does not remove anything from a model already trained, and firewall rules that block crawlers before they ever read robots.txt are invisible from here.
Questions people ask
Should I block AI crawlers?
That is a real choice and not a mistake either way. Blocking training crawlers keeps your writing out of future models, which some people care about a great deal. Blocking search crawlers is different: those are the ones that let an assistant quote you and link to you when somebody asks a question your page answers, and blocking them mostly means being left out of that. The common accident is meaning the first and doing both with one line.
My robots.txt blocks everything. Is that bad?
It depends what the site is for. A staging site or a private tool should block everything. A business website usually should not, and a "Disallow: /" under "User-agent: *" is often left over from before the site launched. The tool shows you the exact line so you can see whether somebody meant it.
Why does one crawler show a different answer from another?
Because a crawler obeys only the group that names it, and ignores the catch-all group entirely if it has one of its own. So a file with a friendly group for GPTBot and a "Disallow: /" under the star blocks everything except GPTBot. That rule surprises nearly everyone, and it is why this tool shows which group each crawler matched.