robots.txt Generator - Build Your Crawler Rules File

How to use this tool

1. Choose a preset. Start from a preset that matches your use case, or build from scratch.
2. Add rules. Each rule block targets a User-agent (use * for all bots). Set Allow/Disallow paths.
3. Add sitemap URLs. Add one or more absolute HTTP(S) sitemap URLs so crawlers can discover them.
4. Download and deploy. Download or copy the generated robots.txt and place it at the root of your web server.

What is robots.txt?

robots.txt is a plain-text file stored at the root of your domain (e.g., https://example.com/robots.txt) that follows the Robots Exclusion Protocol. Web crawlers read this file before visiting your site to understand which pages they are permitted to crawl.

Syntax reference

• User-agent: *: Applies to all crawlers. Use a specific name (e.g., Googlebot) to target one.
• Allow: /path/: Explicitly permits a path that would otherwise be disallowed.
• Disallow: /path/: Prevents the crawler from accessing the path and anything below it.
• Crawl-delay: N: Requests a pause of N seconds between requests (not honoured by Google).
• Sitemap: URL: Points crawlers to your XML sitemap for faster discovery.

AI crawler controls: training, search, and fetching

The AI crawler controls group agents by their published purpose. Training and content-use rules can express an opt-out for crawlers that honor robots.txt. Blocking AI search crawlers can also make your pages less visible in answer engines. A robots.txt rule is voluntary and is not an access-control mechanism.

• GPTBot may be used by OpenAI for foundation-model training.
• ClaudeBot is Anthropic's training crawler.
• Google-Extended controls Google use for Gemini model training and grounding; it does not affect Google Search.
• Applebot-Extended controls foundation-model training use and does not crawl pages itself.
• CCBot collects pages for the Common Crawl dataset.

Search and user-requested fetching use different agents:

• OAI-SearchBot serves ChatGPT search; Claude-SearchBot serves Anthropic search.
• PerplexityBot is for Perplexity search, not training.
• ChatGPT-User, Claude-User, and Perplexity-User support user-directed retrieval or fetching. In particular, robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

Select the agents you want to block in the builder. Each selection adds a separate User-agent group with Disallow: /; clear it to remove that named group. If you only want to opt out of training, leave the search and user-fetch controls unchecked.

Deploying your robots.txt

Place the generated file at exactly /robots.txt on your web server. Most static site generators (Astro, Next.js, Gatsby) let you place a robots.txt in the public/ directory and it will be served automatically.

Robots.txt limitations

The robots.txt protocol is voluntary: well-behaved crawlers (Googlebot, Bingbot) respect it, but malicious scrapers and vulnerability scanners ignore it entirely. To protect genuinely private content, use:

• HTTP authentication — requires a username and password to access the page.
• noindex meta tag — prevents indexing even if a crawler does visit the page.
• Server-side access controls — the only reliable way to block unwanted access.

robots.txt is best used to manage crawl budget (directing crawlers away from low-value pages) not for security.

Sitemap in robots.txt

You can list your sitemap URL(s) in robots.txt to help crawlers discover all indexed pages:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Multiple sitemaps can be listed. All URLs must be absolute (including protocol). While submitting sitemaps directly through Google Search Console is the primary method, the robots.txt declaration helps crawlers from all engines discover them automatically.