Your robots.txt tells search engines and AI crawlers what they may crawl on your store. On Shopify it's mostly automatic, which is good — but the defaults cause two recurring problems, and Google's "Indexed, though blocked by robots.txt" warning trips up a lot of merchants who then make it worse by blocking harder. Here's how the file works on Shopify, how to edit it safely, and how to fix the errors it causes.

How Shopify robots.txt works

Shopify generates the file at yourstore.com/robots.txt automatically and serves a sensible default. That default already disallows the internal paths you never want indexed — /cart, /checkout, /orders, /account, and Shopify's internal search and filtering URLs — and points crawlers to your sitemap. For a lot of stores you never need to touch it.

Since 2021 you can customize it by adding a templates/robots.txt.liquid file to your theme. You create it in the theme code editor: from your admin go to Online Store → Themes → [your theme] → ⋯ → Edit code, then under the Templates folder click "Add a new template" and choose robots.txt from the template type dropdown. Shopify scaffolds the default rules as Liquid, and you edit from there. The file is Liquid, not plain text, so you're modifying a template that renders the rules rather than writing raw directives — which is what lets you add or remove specific rules while keeping Shopify's sensible defaults intact.

Two cautions before you edit. First, this file is powerful in the wrong direction — a stray Disallow: / would tell every crawler to ignore your entire store, so change it deliberately and check the rendered output at yourstore.com/robots.txt afterward. Second, robots.txt controls crawling, not indexing. Those are different things, and confusing them is the root of the most common Shopify robots error.

Crawling is not indexing

A robots.txt Disallow rule tells a crawler "don't fetch this URL." It does not tell Google "don't index this URL." If Google discovers a blocked URL through a link somewhere, it can still add the bare URL to its index — it just can't read the page to see what's on it. The result is a listing with no useful title or description, which is worse than either indexing the page properly or keeping it out entirely.

That single distinction explains the warning below, and it's why "just block it in robots.txt" is the wrong fix for pages you want kept out of search.

Fixing "Indexed, though blocked by robots.txt"

This Search Console warning means Google found a URL, usually via a link, but couldn't crawl it because robots.txt blocks the path — so it indexed the URL with no content. The fix depends on what you want to happen to that URL:

  • Identify the exact blocked URL in Search Console's Pages report under this status, and check it against your robots.txt to find the matching Disallow rule.
  • If it should be indexed — a real product or collection that got caught by an over-broad rule — remove or narrow the Disallow rule in robots.txt.liquid so Google can crawl it, then request indexing. Once Google reads the page, the warning clears and the listing gets a proper title and description.
  • If it should not be indexed — a thin filter page, an internal search result, a login page — don't rely on blocking. Blocking alone can leave the bare URL indexed forever. Instead, let Google crawl the page and give it a noindex signal it can actually read: a noindex robots meta tag in the page markup, or an X-Robots-Tag noindex header. Counterintuitively, to remove a page from the index you often have to unblock it first so the crawler can see the noindex instruction. A blocked page is a page whose noindex tag Google will never read.

Disallowing search and tag pages

Two Shopify URL patterns generate near-infinite low-value pages that can bloat your index: internal search results (paths under /search) and tag-filtered collection URLs (long query strings with ?constraint= or tag parameters). Shopify's defaults handle much of this, but on customized themes and heavily tagged catalogs these can slip through and get indexed as thin, duplicative pages.

If you find them indexed and want them gone, apply the crawl-versus-index logic above. Blocking /search in robots.txt stops fresh crawling but won't remove URLs already indexed; for those, a noindex signal on the search and filter templates is the durable fix. Canonical tags help here too — pointing filtered collection variants back at the clean collection URL is often cleaner than blocking, and it's covered in the canonical tags guide. The judgment call is whether you want the page uncrawlable or unindexed; they call for opposite tools.

Welcome AI crawlers, deliberately

To be cited by AI answer engines, your robots.txt needs to allow the crawlers those engines use — and many stores block them by accident, often through a CDN or security setting rather than the store owner's choice. The agents worth knowing by name:

  • GPTBot and OAI-SearchBot — OpenAI's crawlers, the second specifically for ChatGPT's search citations.
  • ClaudeBot — Anthropic's crawler.
  • Google-Extended — Google's control for AI training and grounding, separate from Googlebot.
  • PerplexityBot — Perplexity's crawler.
  • CCBot — Common Crawl, which feeds many downstream models.

These are distinct from Googlebot and each fetches on its own terms. The single fastest AEO action is confirming your robots.txt does not block the ones you want citing you, and ideally allowing them by name with an explicit User-agent and Allow block in robots.txt.liquid. Blocking them silently opts your store out of AI visibility with no error anywhere to warn you. This is core to answer engine optimization and the wider generative engine optimization and AI visibility work — and it pairs naturally with a good llms.txt file, which is the affirmative version of the same idea. If you're weighing which agents to allow, the ChatGPT SEO guide goes deeper on OpenAI's crawlers specifically.

A fast way to check your robots.txt

Open yourstore.com/robots.txt in a browser — that's the rendered output every crawler sees, and it's the file that matters, not your Liquid template. Scan for any Disallow: / on its own (a full-site block), for AI crawler user-agents sitting under a Disallow, and for a Sitemap: line pointing at your real sitemap.xml. In Search Console, the Pages report tells you what's actually happening — how many URLs sit under "Indexed, though blocked by robots.txt" and which ones. Between the raw file and the Pages report you can see both the policy and its consequences.

Robots.txt hygiene sits alongside your sitemap, redirects, and schema markup as the crawlability layer of Shopify SEO; the 2026 checklist and the Shopify SEO guide tie them together, and the Shopify SEO tools overview covers what automates them.

RankEngine audits your robots policy, flags blocked-but-indexed URLs from your Search Console data, and can publish a robots.txt.liquid with the right AI-crawler rules — then checks it against the live rendered file so you know the change actually took rather than trusting that the template saved.