Skip to content
My Tiny Utils

Search tools

Type to find a tool, then press Enter to open it.

robots.txt Generator

Build a robots.txt from user-agent groups, allow and disallow paths, crawl-delay, and sitemap URLs, with a preset for blocking AI crawlers.

Processed locally in your browser — nothing is uploaded.

Group 1

robots.txt

User-agent: *
Disallow:

About robots.txt Generator

Add a group per user-agent, list the paths to disallow or allow, and the file assembles itself as you type. Sitemap lines go at the end where crawlers expect them, and a group with no rules is written as an explicit "Disallow:" so its intent is unambiguous.

The AI-crawler preset blocks the bots that are currently documented as fetching content for model training or answer engines: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider. Treat it as a starting point — new crawlers appear regularly and each operator publishes its own user-agent string.

Common uses

  • Keep admin, staging, or search-result URLs out of crawl budgets
  • Opt a site out of the AI crawlers that honour robots.txt
  • Point search engines at one or more sitemap files

Example

Input

User-agent * · Disallow /admin · Sitemap https://example.com/sitemap.xml

Output

User-agent: *
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

Limitations

  • robots.txt is advisory — well-behaved crawlers obey it, others ignore it entirely; use authentication for anything private
  • Disallowing a URL does not remove it from an index; that needs a noindex tag or a removal request
  • Googlebot ignores Crawl-delay; set the crawl rate in Search Console instead
  • The AI-crawler list reflects what operators published as of mid-2025 and needs periodic review

Related tools