Robots.txt tester — check if a URL is blocked

Quick answer: paste any URL and this fetches that site robots.txt, tests the path against the major search and AI crawlers, and shows the exact rule that decided each verdict.

Longest match wins, not first match

The rule most people carry around — that Allow and Disallow are applied in the order they appear — is wrong, and it is why a file can look correct and behave otherwise. A crawler selects the single most specific user-agent group that names it, ignores every other group in the file, and then applies the longest matching path within that group. Order does not enter into it.

So this blocks nothing at all:

User-agent: *
Disallow: /collections/
Allow: /collections/

The two patterns are the same length, and a tie goes to Allow.

Crawling is not indexing

Disallow stops a crawler fetching a URL. It does not stop the URL appearing in search results. A blocked page with inbound links can still be listed, as a bare URL with no description, because the engine has been told it may not look at the page it is listing.

If the goal is to keep a page out of the index, the crawl has to be allowed so the noindex directive can be read. Blocking and noindexing the same URL is self-cancelling, and it is a common mistake on thin collection and filter pages.

AI crawlers are separate decisions

GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended each obey their own group. A rule aimed at Googlebot says nothing to any of them, and most sites have taken no position — which is allowed by default, arrived at by not choosing.

Whether that is what you want is a business question rather than a technical one. If you sell something people ask assistants about, being readable is the point; see answer engine optimization.

On Shopify

The default file blocks cart, checkout, account and internal search, which is correct and rarely worth changing. It is editable through the robots.txt.liquid theme file. Our robots.txt generator writes a file; this page checks what a live one actually does.