Skip to content
My Tiny Utils

Search tools

Type to find a tool, then press Enter to open it.

robots.txt Generator

Build a robots.txt from user-agent groups, allow and disallow paths, crawl-delay, and sitemap URLs, with a preset for blocking AI crawlers.

Processed locally in your browser — nothing is uploaded.

robots.txt Generator

Group 1

robots.txt

User-agent: *
Disallow:

About robots.txt Generator

Add a group per user-agent, list the paths to disallow or allow, and the file assembles itself as you type. Sitemap lines go at the end where crawlers expect them, and a group with no rules is written as an explicit "Disallow:" so its intent is unambiguous.

The AI-crawler preset blocks the bots that are currently documented as fetching content for model training or answer engines: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider. Treat it as a starting point — new crawlers appear regularly and each operator publishes its own user-agent string.

Common uses

  • Keep admin, staging, or search-result URLs out of crawl budgets
  • Opt a site out of the AI crawlers that honour robots.txt
  • Point search engines at one or more sitemap files

Example

Input

User-agent * · Disallow /admin · Sitemap https://example.com/sitemap.xml

Output

User-agent: *
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

Limitations

  • robots.txt is advisory — well-behaved crawlers obey it, others ignore it entirely; use authentication for anything private
  • Disallowing a URL does not remove it from an index; that needs a noindex tag or a removal request
  • Googlebot ignores Crawl-delay; set the crawl rate in Search Console instead
  • The AI-crawler list reflects what operators published as of mid-2025 and needs periodic review

Frequently asked questions

Is my site structure sent to a server when I build a robots.txt here?

No. The file is assembled entirely in your browser as you type — the paths, user-agents, and sitemap URLs you enter are never transmitted or stored, so nothing reveals your admin or staging paths to anyone.

Does blocking a page in robots.txt remove it from Google?

No. Disallow only stops crawling; a blocked URL can still appear in results if other pages link to it. To remove a page from the index, allow crawling and add a noindex meta tag, or use Search Console's removal tool.

Which AI crawlers does the block-AI preset cover?

The preset adds a Disallow: / group for GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and Bytespider, plus an allow-everything group for all other user-agents. The list matches what operators had documented as of mid-2025, so review it as new crawlers appear.

Where do I put the robots.txt file on my site?

At the root of the host, so it is reachable at https://example.com/robots.txt. Crawlers only look there — a robots.txt in a subdirectory is ignored, and each subdomain needs its own file.

Does Crawl-delay actually work?

Only for some crawlers. Bing and Yandex honour Crawl-delay as seconds between requests, but Googlebot ignores the directive entirely — Google's crawl rate is managed automatically or via Search Console.

Related tools