Robots.txt Generator — with AI Crawler (GPTBot) Presets
DeveloperA robots.txt generator combines a crawl policy, Disallow paths, a sitemap location, and AI-crawler allow/block choices into a robots.txt file ready for your site root.
What is it
robots.txt is a plain-text file at the site root (/robots.txt) that declares per-crawler (User-agent) Allow and Disallow rules plus the sitemap location, following the RFC 9309 Robots Exclusion Protocol. This tool builds the file from a base policy and blocked paths, and adds presets for well-known AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, and CCBot. Because compliance is voluntary, robots.txt is not a security mechanism.
How to use
- 1Choose the base policy (allow all or disallow all) and list paths to block.
- 2Pick the AI crawler preset and add your sitemap URL.
- 3Download the generated robots.txt and upload it to the site root (/robots.txt).
Reference
| Crawler | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | AI model training crawl |
| ChatGPT-User | OpenAI | live fetches for ChatGPT users |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ClaudeBot | Anthropic | AI model training crawl |
| Google-Extended | Gemini training control token | |
| PerplexityBot | Perplexity | AI search index |
| CCBot | Common Crawl | open web archive (reused for AI training) |
| Bytespider | ByteDance | AI training crawl |
| Applebot-Extended | Apple | Apple AI training control token |
Sources & standards
- RFC 9309: Robots Exclusion Protocol - IETF
- Introduction to robots.txt - Google Search Central
- OpenAI crawlers and bots - OpenAI
FAQ
Are my rules sent to a server?
No. The robots.txt is generated entirely in your browser; nothing is uploaded.
Does blocking AI crawlers in robots.txt fully stop them?
No. robots.txt relies on voluntary compliance (RFC 9309). Major AI crawlers such as GPTBot state that they honor it, but bots that ignore the rules can only be stopped at the server or firewall level.
Does Disallow remove pages from search results?
No. Disallow only prevents crawling; already-known URLs can still be indexed. Use a noindex meta tag when the goal is removal from search results.