Robots.txt, Sitemaps and Indexing

These tools manage what search engines can crawl and index. Write or test a robots.txt file, build, find or validate a sitemap, and check whether a noindex directive or a canonical tag points a page somewhere unexpected.

Tools in this group

• Canonical Tag Checker — Enter a URL, paste its source, or both, to check which canonical URL the page declares.
• Noindex Checker — Enter a URL and paste its source to see whether anything stops Google from indexing it.
• Robots.txt Example Files — Pick a robots.txt example to see the file, what each line means and which paths it blocks.
• Robots.txt Generator — Choose what crawlers may visit to get a ready robots.txt file, with a test of what it blocks.
• Robots.txt Tester — Fetch a site's robots.txt, or paste one, and test which URLs a crawler may visit.
• Sitemap Checker and Validator — Paste a sitemap or open the file to check it against the sitemap protocol, with line numbers.
• Sitemap Finder — Enter a website to find the sitemaps its robots.txt lists and the usual places to look.
• XML Sitemap Generator — Paste your page URLs to get a valid sitemap.xml file, or several files and an index.