AI Visibility · free tool

Free llms.txt Checker

Enter a URL to check for an llms.txt file, inspect your robots.txt rules for the major AI crawlers (GPTBot, ClaudeBot, PerplexityBot), and probe whether your CDN or firewall blocks AI bots even when robots.txt allows them. See whether AI search engines can read and cite your site.

Last updated 11 November 2026
Runs the same check as a full AuditZap audit
No signup required. Results in seconds
We don't store the URLs you check

What this llms.txt checker checks

This free llms.txt checker looks for an llms.txt file on the domain you enter and inspects your robots.txt rules for 16 well-known AI crawlers, split into the search and citation agents that answer live questions (OAI-SearchBot and ChatGPT-User for OpenAI, PerplexityBot and Perplexity-User, Claude-SearchBot and Claude-User for Anthropic, Google-CloudVertexBot for Vertex AI Search, and DuckAssistBot for DuckDuckGo), the dual-use Amazonbot (Alexa and Rufus answers, also used for training), and the training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, and Google-Extended, which gates Gemini training and Gemini/Vertex grounding only, not Google Search). It tells you whether AI search engines and assistants are allowed to read your site and whether you are publishing an llms.txt to guide them.

This is about generative engine optimisation (GEO): being visible and citable in AI answers, not just the ten blue links.

llms.txt validator: what we validate

This tool is an llms.txt validator as well as a checker. It checks four things:

  • We check the H1 format. The file must be plain markdown whose first line is a # H1 title, as the llmstxt.org format asks. An empty file, or an HTML page served at /llms.txt, fails.
  • We check that links resolve. Once the format passes, we request up to 10 links from the file and list each one as works, broken, or could not check. A link that answers with a security challenge is never counted as broken.
  • We check AI crawler access. We read your robots.txt rules for 16 AI crawlers to see which of them may read your site.
  • We check for CDN blocking. We request your page as a browser and as GPTBot and compare the answers, to catch a CDN or firewall that turns AI crawlers away.

Your llms.txt score

The score out of 100 covers the file itself: 30 points for a readable text file, 30 for a # H1 first line, and 40 for the share of checked links that work. Links we could not check do not lower it. If the file could not be fetched, or there is no file, you get no score rather than a low one.

No file yet? The llms.txt Generator builds one from your key pages. Publish it at your site root, then validate it here.

Why AI crawler access matters

A growing share of search now happens inside AI assistants and answer engines that summarise the web and cite their sources. If your robots.txt blocks GPTBot, ClaudeBot, or PerplexityBot, those systems cannot read your content and will not cite you, so you lose visibility in a fast-growing channel.

An llms.txt file is an emerging convention: a plain-text file at your root that points AI crawlers at your most important, cleanest content, much as a sitemap does for search engines. It is optional today but a cheap way to signal what matters.

When robots.txt says yes but your CDN says no

robots.txt is only a request. Many CDNs and firewalls, Cloudflare among them, now block AI crawlers at the network level on their default settings, so a request identifying as GPTBot gets an HTTP 403 even though your robots.txt welcomes it. Every robots.txt-only checker misses this, because the block never touches robots.txt.

This checker probes your URL twice, once as a normal browser and once as GPTBot, and compares the two. If the browser request succeeds but the bot request is blocked, your CDN or firewall is the thing turning AI crawlers away, and no robots.txt edit will fix it. You will need to allow the verified AI crawlers in your CDN bot-management settings instead.

How to open up (or control) AI access

Decide deliberately whether you want AI systems reading your site, then make robots.txt match that intent:

  • To be citable: make sure robots.txt does not Disallow GPTBot, ClaudeBot, PerplexityBot, and similar agents.
  • To opt out: add explicit Disallow rules for the AI user-agents you do not want crawling you.
  • Publish an llms.txt at your root linking to your key pages and documentation to help AI systems find your best content.

Honest limits

This tool checks for llms.txt, validates its format and links, and inspects robots.txt rules for the well-known AI crawlers on the domain you enter. It cannot guarantee a given AI system will or will not crawl you, since compliance is voluntary. To confirm your robots.txt is not also blocking Google itself, pair it with the robots.txt checker. For the full set of AI Visibility (GEO) checks alongside your traditional SEO, run a full free audit: all 40 checks with full detail plus one AI fix, across up to 10 pages. Pro adds an AI fix for every issue, a PDF report and up to 100 pages.

Full audit

Run all 40 checks on this site

Every check on up to 10 pages free, with the fix written for your platform.

Run free audit

Frequently asked questions

Is this llms.txt checker free?

Yes, free with no signup, up to 15 checks per hour.

What is llms.txt?

It is an emerging plain-text file at your site root that points AI crawlers at your most important content, similar in spirit to a sitemap for search engines. It is optional today, but a low-effort way to guide AI systems to your best content.

Which AI crawlers does it check?

It inspects robots.txt for 16 agents: search and citation bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Google-CloudVertexBot, DuckAssistBot), the dual-use Amazonbot, and training crawlers (GPTBot, ClaudeBot, CCBot, Applebot-Extended, Meta-ExternalAgent, Bytespider, Google-Extended).

Why would AI crawlers be blocked if my robots.txt allows them?

Because robots.txt is only one layer. Your CDN or firewall (Cloudflare, for example) can block AI crawlers at the network level regardless of robots.txt, returning a 403 to GPTBot while a browser loads fine. This checker probes as both a browser and GPTBot to catch that mismatch, then you fix it in your CDN bot settings, not robots.txt.

Is this an llms.txt validator?

Yes. Besides finding the file, it checks that the file is readable markdown with a # H1 title on its first line and that its links work, then scores it out of 100: 30 points for a readable text file, 30 for the H1 title and 40 for the share of checked links that work. If we cannot fetch the file, or there is no file, you get no score rather than a low one.

Does it check the links inside llms.txt?

Yes, up to 10 links per file, in the order they appear, once the file passes the format check. Each one is listed as works, broken (for example a 404), or could not check. A link that answers with a security challenge (HTTP 401, 403, 407 or 429) or does not respond is marked could not check and does not lower your score. Links to private or local addresses are skipped.