Free Tool · Crawlers
Free robots.txt checker
for AI crawlers.
Free to use — this tool earns us no commission or referral fee. We may invite you to talk about paid work afterward; that is optional and labelled separately.
Paste a public URL. See which robots.txt rule applies to that page for each search or training crawler. This checks declared access; it does not predict rankings or AI citations.
Check a public origin
No account · No cookies · 2 checks per day
| Bot | Group | Access | Matching rule |
|---|
Raw robots.txt
Frequently Asked Questions
What does this robots.txt checker look at?
It evaluates robots.txt rules for the exact URL you enter, including named crawler groups, wildcard fallback and the longest matching Allow or Disallow rule. It cannot test firewall access or confirm indexing.
Does blocking GPTBot hide me from ChatGPT search?
GPTBot controls potential training use. OAI-SearchBot handles OpenAI search crawling and ChatGPT-User handles user-initiated requests. They are separate controls: blocking GPTBot is not the same as blocking ChatGPT search.
Why is there a daily limit?
Each check fetches a live page and its robots.txt through the same server-side function as the GEO audit. Two free checks per day are recorded in your browser, with an additional network limit on the server.
Do you store the URL I enter?
The URL is sent to our server to fetch the public page and robots.txt. We do not save an audit history. Daily usage counts are stored in your browser and a temporary network identifier is used for server rate limiting. Do not enter private or sensitive URLs.