Free Robots.txt Checker
Enter a URL to fetch and analyse its robots.txt file. We'll flag any Disallow rules that block Googlebot from crawling important pages.
What this robots.txt checker checks
This free robots.txt checker fetches the robots.txt file from the domain you enter and inspects it for Disallow rules that block search engines from crawling important pages. A single overly broad rule (such as Disallow: /) can wipe a site out of Google, and it is one of the most common and most damaging technical SEO mistakes.
For the fuller picture of how robots.txt and sitemaps work together, read the robots and sitemap guide.
Why robots.txt matters
Robots.txt is the first file crawlers read on your site, and it tells them where they may and may not go. Get it right and crawlers spend their budget on the pages that matter. Get it wrong and you can accidentally block your whole site, a key section, or your CSS and JavaScript, which stops Google rendering pages correctly.
The classic disaster is a staging-site robots.txt with Disallow: / going live unchanged, silently deindexing the production site. A quick check catches it before rankings collapse.
How to fix robots.txt problems
Keep the file minimal and deliberate:
- Remove any Disallow: / rule on production unless you genuinely want the whole site hidden.
- Do not block CSS, JavaScript, or image directories that Google needs to render your pages.
- Use robots.txt to keep crawlers out of admin, cart, and search-result URLs, not to hide pages from the index (use a noindex tag for that).
- List your sitemap URL in robots.txt so crawlers can find it.
Honest limits
This tool fetches and analyses the live robots.txt for the domain you enter and flags blanket-blocking rules. It does not test every individual URL against every user-agent rule. To see robots.txt findings alongside indexability, sitemaps, and the rest of your technical SEO in a PDF report, run a full free audit.
Frequently asked questions
Is this robots.txt checker free?
Yes, free with no signup, up to 15 checks per hour.
What does Disallow: / do?
It tells crawlers not to crawl any URL on the site, which can remove the entire site from search. It is almost never what you want on production.
Should I block pages with robots.txt?
Robots.txt controls crawling, not indexing. To keep a page out of the index, use a noindex meta tag instead; a blocked page can still be indexed without its content.
Should I list my sitemap in robots.txt?
Yes. Adding a Sitemap: line helps crawlers discover your sitemap. See the robots and sitemap guide.
Can I check the rest of my technical SEO?
Run a full free audit for indexability, sitemaps, and 38 other checks in one report.