Fetch and parse robots.txt for any domain. See allow/disallow rules per user-agent, crawl delays, and sitemap directives.