Set your crawl rules, tick the AI crawlers you want to block, add your sitemap, and copy or download a correct robots.txt.
How it works
The generator builds the file live from your choices: a default rule group for all bots (allow everything, block everything, or your custom Disallow/Allow paths), one explicit block per AI crawler you tick, and the Sitemap line. Copy it, or download robots.txt ready to upload to your site root.
Honest limits: robots.txt is a request that well-behaved crawlers honor, not access control, and blocking a page from crawling does not remove it from the index (use a noindex meta tag for that). Everything runs in your browser.
Frequently asked questions
What does robots.txt do?
It tells crawlers which parts of your site they may fetch. Well-behaved bots (Google, Bing, the AI crawlers listed here) read it before crawling and honor the rules. It lives at exactly yoursite.com/robots.txt.
Does robots.txt hide pages from Google?
No, and this matters: robots.txt blocks CRAWLING, not INDEXING. A blocked page can still appear in results (without a description) if other sites link to it. To keep a page out of the index, allow crawling and use a noindex robots meta tag instead. Never rely on robots.txt for private content, it is a public request, not access control.
How do I block AI crawlers like GPTBot and ClaudeBot?
Tick the bots you want to block (or the master toggle) and the generator emits a User-agent block with Disallow: / for each: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, and meta-externalagent. These vendors state they honor robots.txt; blocking Google-Extended opts out of Gemini training without affecting Google Search.
What is the difference between Disallow and Allow?
Disallow blocks paths that start with the value; Allow carves exceptions back out (e.g. disallow /admin/ but allow /admin/public/). An empty Disallow: means "everything allowed", which is the correct explicit default.
Why no crawl-delay option?
Honestly, because Google ignores it, it was never standardized, and misuse hurts crawling more than it helps. Set crawl rate in each engine's webmaster tools instead.
Where do I put the file?
Download it and upload to your site root so it is served at https://yoursite.com/robots.txt. Subdirectories do not work; each subdomain needs its own file. Then test it with our Robots.txt Tester.