Robots.txt generator

Build a valid robots.txt file to tell search engines and crawlers which parts of your site they can fetch. Start from a preset, tweak the rules, then copy or download. Everything runs in your browser.

Quick presets
Sitemaps
robots.txt
User-agent: *
Disallow:

robots.txt is advisory. Well-behaved crawlers respect it, but it is not a security control and must live at your domain root (for example https://example.com/robots.txt).

Once your robots.txt is live, keep an eye on it. SiteGauge alerts you if it changes so an accidental edit never quietly deindexes your site.Monitor this

How it works

  • Pick a preset to get started, then add or edit rules. Each group targets one User-agent (use * for all crawlers) and lists the paths to Allow or Disallow.
  • A Disallow with an empty path means "allow everything"; Disallow: / blocks the whole site. Paths are matched from the domain root, so /admin covers /admin and everything under it.
  • Add Sitemap lines so search engines can find your XML sitemaps, and an optional Crawl-delay to slow aggressive bots (Google ignores it; Bing and others honour it).
  • Copy the result or download a ready-to-upload robots.txt. Place it at your domain root so it is reachable at /robots.txt.