Free Tool · Crawlers
Free robots.txt checker
for AI crawlers.
Free to use — this tool earns us no commission or referral fee. We may invite you to talk about paid work afterward; that is optional and labelled separately.
Paste a public URL. We fetch robots.txt and show whether named bots are allowed, blocked, or falling through to * — and whether each named group has its own Content-Signal.
Check a public origin
No account · No cookies · 2 checks per day
| Bot | Group | Access | Content-Signal |
|---|
Raw robots.txt
What are the common questions about Frequently Asked Questions?
What does this robots.txt checker look at?
It fetches robots.txt for the origin of the URL you enter and maps named AI crawlers (GPTBot, Google-Extended, PerplexityBot, ClaudeBot, Applebot, bingbot and others) to Allow/Disallow and Content-Signal. Named groups do not inherit signals from User-agent: *.
Does blocking GPTBot hide me from ChatGPT search?
It can hide training and some retrieval paths. ChatGPT-User and OAI-SearchBot are separate agents. This checker shows each group; it does not tell you which product will still fetch you. No ranking promise.
Why is there a daily limit?
Each check fetches a live page and its robots.txt through the same server-side function as the GEO audit. Two free checks per day, stored in your browser.
Do you store the URL I enter?
No. Rate-limit counts live only in local storage. This tool earns us no commission.