Robots.txt Generator

Build a valid robots.txt with allow/disallow rules and sitemap.

Upload the file to your site root so it is reachable at https://yoursite.com/robots.txt.

What Does the Robots.txt Generator Do?

This generator builds a correct robots.txt file from simple form choices — no syntax to memorize. Set a default policy (allow or block everything), list paths to disallow, add exceptions, point crawlers to your sitemap, set an optional crawl-delay, and with one checkbox block AI training crawlers (GPTBot, CCBot, Google-Extended) or image indexing. The finished file is ready to copy or download.

robots.txt is the first file well-behaved crawlers request from your site. It lives at the domain root and tells search engines which areas they may crawl — getting it wrong can accidentally hide your whole site from Google.

How to Create Your Robots.txt

  1. Choose the default policy for all robots.
  2. List paths to block, one per line (e.g. /admin/, /cart/).
  3. Add your sitemap URL — strongly recommended for faster indexing.
  4. Optionally block AI crawlers or image search.
  5. Click Generate, then upload the file to your site root so it is reachable at yoursite.com/robots.txt.

Rules Worth Knowing

  • robots.txt is not security — it is a polite request. Malicious bots ignore it, and blocked URLs can still appear in results if linked elsewhere. Use authentication or noindex for genuinely private content.
  • Blocking a page ≠ de-indexing it — to remove a page from Google, allow crawling and add a noindex meta tag instead.
  • Paths are prefix matches — Disallow: /private blocks /private/, /private.html and /private-anything.
  • One file per host — subdomains need their own robots.txt.

Frequently Asked Questions

Where do I put the robots.txt file?

In the root of your domain, so it loads at https://yoursite.com/robots.txt. Search engines only look there — a robots.txt in a subfolder is ignored.

Does Disallow keep a page out of Google entirely?

Not necessarily. Disallow prevents crawling, but a URL that other sites link to can still be indexed without content. To fully remove a page, let it be crawled and add a noindex meta tag.

What does Crawl-delay do?

It asks crawlers to wait N seconds between requests, easing load on small servers. Bing and Yandex respect it; Googlebot ignores it (use Search Console's crawl-rate settings instead).

Should I block AI crawlers like GPTBot?

That is a policy choice. Blocking GPTBot, CCBot and Google-Extended asks AI companies not to use your content for model training, without affecting normal search indexing.