AI Crawler Policy Generator
FeaturedBuild robots.txt rules for GPTBot, Google-Extended, ClaudeBot and more.
SEO ToolsFree & Online
Loading tool...
About this tool
Writes robots.txt groups for major AI crawlers. Search bots (Googlebot) stay separate from training bots (GPTBot, Google-Extended, ClaudeBot, CCBot).
Presets: allow all, block training but keep retrieval, or block AI while allowing Google Search.
How to use
- Pick a preset or toggle each bot.
- Keep User-agent * allowed unless the site should not be crawled at all.
- Download robots.txt to the site root.
Features
GPTBot and ChatGPT-User
Google-Extended vs Googlebot
ClaudeBot / anthropic-ai
Perplexity, CCBot, Bytespider
Training vs retrieval presets
Sitemap line