What does this tool check?
It fetches the robots.txt file for a site and checks it for common mistakes. Then it fetches every sitemap the file lists, or /sitemap.xml if none is listed, validates the XML and counts the URLs. Finally, it requests a small sample of the listed pages to see whether they still work.
What mistakes does it find in robots.txt?
It flags a Disallow: / that blocks the whole site, rules that come before any User-agent line, unknown or misspelled directives, directives that Google and Bing ignore, sitemap lines that are not full URLs, a missing Sitemap line, and a file over 500 KB, which Google stops reading.
How does the crawl test work?
Pick a crawler name, such as Googlebot, and a page. The tool applies the file's rules the way a crawler would: it picks the most specific matching User-agent group, then the longest matching Allow or Disallow rule. It then tells you which rule decided the result.
What are the sitemap limits?
One sitemap file can list up to 50,000 URLs and be up to 50 MB uncompressed. A site with more than that needs several sitemaps listed in a sitemap index. Files too large to download here are reported, not checked in full.
Why check the sample URLs?
A sitemap should only list pages that return 200. Listing URLs that redirect or return 404 wastes crawl effort and can make search engines trust the file less. The tool spot-checks up to five URLs, spread evenly through the list, and warns if any are blocked by robots.txt.
Does a missing robots.txt cause problems?
Not by itself. If the file returns 404, crawlers treat the whole site as open. A server error (5xx) is different: Google treats it as if the whole site is blocked until the file works again.
Can I check any website?
Yes, as long as it is publicly reachable over HTTP or HTTPS. Local, private and internal network addresses are not supported.
What about links on the pages themselves?
This tool looks at the two files that guide crawlers. To check every link on a site, use the link crawler. For security headers, SSL and meta tags, run the site audit.