Fetch any site's robots.txt and check which URLs are allowed or blocked for any crawler, Googlebot to GPTBot, with the exact rule that decided it.
robots.txt
Test URLs against it
How it works
The tester fetches the site's live robots.txt once, then everything else happens in your browser: the file is parsed per the robots.txt standard (RFC 9309) and each URL you enter is matched exactly the way crawlers do it, most specific user-agent group, longest rule wins, Allow beats Disallow on ties, with * and $ wildcards honored. Every verdict shows the rule and line number responsible.
The crawler list includes today's AI training and browsing bots, so "did we actually block GPTBot?" is a ten-second check. You can also paste a draft file and test it locally before deploying.
Frequently asked questions
How do I test a robots.txt file?
Enter the domain and click Fetch robots.txt. The live file appears, then you pick which crawler to test as (Googlebot, GPTBot, ClaudeBot, and more, or a custom token), paste the URLs you care about, and each one comes back Allowed or Blocked with the exact rule and line number that decided it.
How are the rules interpreted?
The way the robots.txt standard (RFC 9309) and Google interpret them: the crawler uses the most specific matching user-agent group, the longest matching rule path wins, Allow beats Disallow on a tie, * matches anything inside a path and $ anchors the end, and a missing file means everything is allowed.
Can I check whether AI crawlers are blocked?
Yes, that is built in: test as GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, or meta-externalagent and see exactly what each may fetch. To generate the blocking rules, use our Robots.txt Generator.
The site has no robots.txt. What does that mean?
Everything is allowed: crawlers treat a 404 on robots.txt as "no restrictions". The tester reports this honestly instead of erroring. Note some sites return an HTML page at /robots.txt (single-page-app fallbacks); crawlers ignore that, and so do we, flagging it for you.
Does Blocked mean the page is out of Google?
No, blocked means Google will not CRAWL it; the URL can still be indexed from links (shown without a description). To remove a page from results, allow crawling and use a noindex meta tag. This distinction is the most common robots.txt mistake.
Can I test a robots.txt I have not deployed yet?
Yes, click "paste a robots.txt instead" and drop your draft in; parsing and testing run entirely in your browser with no fetch. Perfect for validating a file from our Robots.txt Generator before uploading it.
Are there limits?
Fetching a live robots.txt uses one unit of your daily page-fetch allowance (the quota banner shows it); pasted files are unlimited since nothing touches the server. Up to 100 URLs test at once.