Robots.txt generator
Build a valid robots.txt file to tell search engines and crawlers which parts of your site they can fetch. Start from a preset, tweak the rules, then copy or download. Everything runs in your browser.
robots.txt is advisory. Well-behaved crawlers respect it, but it is not a security control and must live at your domain root (for example https://example.com/robots.txt).
How it works
- Pick a preset to get started, then add or edit rules. Each group targets one User-agent (use
*for all crawlers) and lists the paths toAlloworDisallow. - A
Disallowwith an empty path means "allow everything";Disallow: /blocks the whole site. Paths are matched from the domain root, so/admincovers/adminand everything under it. - Add
Sitemaplines so search engines can find your XML sitemaps, and an optionalCrawl-delayto slow aggressive bots (Google ignores it; Bing and others honour it). - Copy the result or download a ready-to-upload
robots.txt. Place it at your domain root so it is reachable at/robots.txt.
What to do with the results
- Double-check the file replaced, not appended to, any existing robots.txt. A stray old rule left in place alongside your new one can block a path you meant to allow.
- Never rely on Disallow to hide something private. It only asks crawlers not to fetch a path; the file itself is publicly readable, so use real authentication for anything genuinely sensitive.
- Blocking a path does not deindex it. A disallowed URL can still appear in search results if other sites link to it. To actually keep a page out of the index, let it be crawled and add a noindex tag instead.
- Test before you rely on it. Fetch the live file at
/robots.txtafter upload and check the rules read the way you built them, since a stray typo in a path can block far more than intended.
One wrong edit to a live robots.txt can quietly deindex a section of a site with no obvious symptom. Ongoing SEO monitoring can watch the file and alert you the moment it changes, so an accidental edit gets caught fast rather than months later.
Frequently asked questions
Start from a preset (allow everything, block everything, WordPress or Shopify), adjust the Allow and Disallow rules in each user-agent group, add your sitemap URL, then copy or download the finished file. Upload it to your domain root so it is served at /robots.txt; that exact location is the only place crawlers look for it. The preview updates as you edit, so what you see is exactly the file you ship.
Disallow stops compliant crawlers fetching a path; it does not remove pages from search results, and a disallowed URL can still be indexed without content if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex meta tag or header instead. Combining the two is a classic mistake: if the path is disallowed, the crawler never fetches the page, so it never sees the noindex.
Add a separate group for each bot: a User-agent line naming it (GPTBot, ClaudeBot, CCBot, Google-Extended) with Disallow: / beneath. A named group overrides the * group for that bot, so your rules for normal search crawlers stay untouched. Whether to block is a trade-off between controlling content reuse and staying visible in AI-generated answers, so decide it as a business question rather than a purely technical one.
Google ignores Crawl-delay entirely; its crawl rate is managed through Search Console. Bing, Yandex and various smaller crawlers do honour it as a minimum number of seconds between requests. Leave it empty unless a specific bot is genuinely hammering your server, because a large value on a big site can slow legitimate crawling enough to delay indexing.
No. It is a publicly readable, advisory file: well-behaved crawlers respect it, and anything else can read it as a signpost to the paths you least want visited. Never list secret URLs in it. Use authentication for anything genuinely private, and noindex for pages that simply should not appear in search results.