← All tools
Robots.txt Checker
Fetch and preview robots.txt for any domain, including sitemap directives.
What this tool checks
This tool fetches the robots.txt file from the domain root and displays its contents. It shows user-agent rules, disallow directives and any sitemap references included in the file.
Why it matters
A misconfigured robots.txt can block search engines from indexing important pages — or accidentally expose private URLs. Regularly reviewing it ensures crawlers can reach your content.
How to fix common issues
Check that you are not blocking important pages with Disallow rules. Make sure your sitemap URL is listed. Use 'Disallow: /admin' style rules only for pages you genuinely want hidden from search engines. Test changes in Google Search Console's robots.txt tester.
Frequently asked questions
- What is robots.txt?
- robots.txt is a plain text file at the root of a website that tells search engine crawlers which pages or directories they should or should not access.
- Can robots.txt block indexing?
- robots.txt can prevent crawling, but it does not guarantee a page won't be indexed. For true noindex, use a meta robots tag or X-Robots-Tag header.
- Does this tool check all crawler rules?
- This tool displays the full robots.txt content so you can review all User-agent and Disallow rules. It also highlights sitemap references.