RankEngineQuick answer: pick your platform and how you want classic search, AI training bots, and AI answer engines treated — the generator writes a correct robots.txt with per-bot rules and a Content-Signal line. For a store, the usual right answer is allow everything: blocking AI crawlers removes you from the answers shoppers are already reading.
Free tool
Free Robots.txt Generator — AI Crawlers, Shopify & More
The decision that actually matters in 2026
The old robots.txt questions — block /cart, point at the sitemap — are solved defaults. The live decision is AI access, and it has two halves most generators conflate:
- Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content to train future models. Blocking them is a philosophical stance with a commercial cost: models that never read your store cannot recommend it from memory.
- Answer-engine crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) fetch pages live to answer a user’s question right now — often "what should I buy". Blocking these removes you from answers at the exact moment of purchase intent.
For a store, allowing both is almost always right: AI answers are a growing discovery channel, and robots.txt is where you either show up or vanish. Publishers with licensing leverage face a different calculus — but a merchant’s product pages benefit from every engine that can read them. The full reasoning is in our AI search optimization guide and the Shopify robots.txt guide.
One more thing robots.txt cannot do: make you citable. That takes structured data, an llms.txt manifest, and content engines can lift answers from — the layer RankEngine automates for Shopify stores.
Frequently asked questions
What does robots.txt actually do?
It tells crawlers which paths they may fetch. It does not remove pages from the index — a blocked URL can still be indexed from external links, just without content. To keep a page out of results, use a noindex meta tag and let crawlers see it. robots.txt is for managing crawl access, not visibility.
Should I block AI crawlers like GPTBot and ClaudeBot?
For an online store, usually no — blocking AI crawlers makes your products invisible to ChatGPT, Claude, and Perplexity exactly when shoppers are asking them what to buy. The nuanced move this generator supports: distinguish training crawlers (GPTBot, CCBot) from answer-engine crawlers (OAI-SearchBot, PerplexityBot) and decide each policy deliberately.
How do I edit robots.txt on Shopify?
Shopify serves an auto-generated robots.txt you cannot edit directly. To customize it, create the robots.txt.liquid template: Online Store → Themes → Edit code → Add a new template → robots.txt. Paste rules there — Shopify’s defaults (cart, checkout, admin blocks) remain unless you override them, so only add what you want to change.
What is the Content-Signal line?
An emerging convention (promoted by Cloudflare) that declares how content may be used: search=yes/no, ai-train=yes/no, ai-input=yes/no. Crawlers are beginning to respect it as a machine-readable usage policy alongside per-bot rules. This generator writes it to match the choices you make above.
Where does the file go?
At the root of the domain: yourstore.com/robots.txt. Search engines only check that exact path. On Shopify the robots.txt.liquid template handles placement automatically; on other platforms upload it to the web root.
More free SEO tools
SEO Title Generator & Checker
Generate SEO titles from your keyword and check any title’s character count, pixel width, and keyword position.
Meta Description Generator & SERP Preview
Generate high-converting meta descriptions and preview exactly how your page looks in Google.
Keyword Density Checker
Paste any text and see keyword density, top 1–2–3-word phrases, and stuffing warnings.
LSI & Related Keyword Generator
Generate related keywords, question keywords, and commercial long-tails from any seed term.
Image Alt Text Generator & Checker
Build descriptive, SEO-friendly alt text for product images and check existing alt text for length, redundancy, and stuffing.
Googlebot Simulator — See What Google Sees
Fetch any URL with Googlebot’s user agent and see exactly what Google sees: status code, redirects, title, meta robots, canonical, X-Robots-Tag, H1s, and an indexability verdict.
Related guides
- ai.txt vs llms.txt: Which File Do AI Crawlers Read?
- llms.txt generator for Shopify (free): how to create one
- How to Get Indexed by LLMs Through an llms.txt File
- Shopify robots.txt: edit and fix it right
Run this across your whole catalog
RankEngine applies and verifies these fixes on every product in your Shopify store — automatically.
Install RankEngine free