Free · live registry data · no login

Robots.txt & Sitemap Checker

Validate crawl rules and sitemap reachability.

How it works

Check robots.txt and sitemap.xml for crawl problems

Enter a domain and we fetch /robots.txt and /sitemap.xml from our server, plus any sitemap URLs the robots file declares. The robots file is parsed into user-agent groups with their Allow, Disallow and Crawl-delay rules; the sitemap is checked for a valid XML root, URL count, lastmod coverage and whether it is a sitemap index. We flag the mistakes that silently remove sites from Google: a blanket Disallow: /, a sitemap that returns HTML or 404, and a robots file with no Sitemap line.

Results appear here after you click Check robots & sitemap.

How this works

1
Fetch both files
Our server requests /robots.txt and /sitemap.xml over HTTPS in parallel, then any extra sitemap URLs robots.txt declares.
2
Parse the rules
robots.txt is split into user-agent groups with Allow, Disallow, Crawl-delay and Sitemap lines; sitemap XML is checked for its root element and counted.
3
Flag what blocks crawling
Blanket disallows, 5xx responses, HTML masquerading as XML, missing Sitemap lines and oversized files are called out with fixes.

Frequently asked questions

How do I check if my robots.txt is blocking Google?: Open yourdomain.com/robots.txt and look for a group with User-agent: * or User-agent: Googlebot followed by Disallow: /. That single line blocks the whole site. This tool parses the file and shows exactly which paths each crawler is barred from, and Google Search Console has a robots.txt report with the last version it fetched.
Where should my sitemap be and how do crawlers find it?: Convention is /sitemap.xml at the root, but any URL works if you declare it with a Sitemap: line in robots.txt and submit it in Google Search Console and Bing Webmaster Tools. Sites with more than 50,000 URLs use a sitemap index that points to several child sitemaps.
Does robots.txt stop a page from appearing in Google?: Not reliably. Disallow stops crawling, but if other sites link to the page Google can still index the URL without content. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.
What happens if robots.txt returns a 500 error?: Google treats a server error on robots.txt as 'assume everything is disallowed' and will stop crawling the site, retrying for about 30 days before giving up and treating it as missing. A 404 is harmless; a 5xx is an emergency.
Why does my sitemap show fewer URLs than my site has?: Many generators exclude noindex pages, paginated archives and drafts by design. If real pages are missing, check the plugin settings or regenerate it. We read the first 64 KB of the file, so for very large sitemaps the count shown is a lower bound.

Every MailShrine domain includes WHOIS privacy, transfer lock, Anycast DNS with DNSSEC and two business mailboxes.