Robots.txt Checker

It fetches the robots.txt file from a domain (always located at the root, e.g. example.com/robots.txt) and parses it into readable rules. You see each user-agent group, its Allow and Disallow paths, any Crawl-delay value, and the sitemap URLs declared in the file. This tool shows both a parsed breakdown and the raw file so you can confirm what crawlers are told to do.

Disallow tells a crawler not to request the matching path, while Allow explicitly permits a path, typically to carve out an exception inside a broader Disallow. For example, Disallow: /admin/ blocks that folder, and a separate Allow: /admin/public/ can re-open one part of it. An empty Disallow value (Disallow: with nothing after it) means nothing is blocked, so the whole site is crawlable for that user-agent.

No. Robots.txt only requests that crawlers skip fetching a URL; it does not remove the page from search results. A blocked URL can still get indexed (usually with no description) if other sites link to it, because Google can index the URL without crawling its content. To actually keep a page out of the index, let it be crawled and use a noindex meta tag or an HTTP header instead of blocking it in robots.txt.

The Sitemap directive points crawlers to your XML sitemap so they can discover your URLs more efficiently. It is optional but recommended, must be an absolute URL (starting with https://), and you can list more than one. This checker pulls out every Sitemap entry it finds so you can confirm the link is present and correct.

Look through the parsed user-agent groups for names like GPTBot, ClaudeBot, CCBot, or Google-Extended, and check whether their rules Allow or Disallow your paths. A rule under User-agent: * applies to crawlers that do not have their own named group, so a bot is only specifically blocked if it appears by name with a Disallow. Since these rules change often, SiteGauge can monitor your robots.txt over time and alert you the moment an AI crawler is newly blocked or allowed.