RankEngineQuick answer: pick your platform and how you want classic search, AI training bots, and AI answer engines treated — the generator writes a correct robots.txt with per-bot rules and a Content-Signal line. For a store, the usual right answer is allow everything: blocking AI crawlers removes you from the answers shoppers are already reading.
Free tool
Free Robots.txt Generator — AI Crawlers, Shopify & More
The decision that actually matters in 2026
The old robots.txt questions — block /cart, point at the sitemap — are solved defaults. The live decision is AI access, and it has two halves most generators conflate:
- Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content to train future models. Blocking them is a philosophical stance with a commercial cost: models that never read your store cannot recommend it from memory.
- Answer-engine crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) fetch pages live to answer a user’s question right now — often "what should I buy". Blocking these removes you from answers at the exact moment of purchase intent.
For a store, allowing both is almost always right: AI answers are a growing discovery channel, and robots.txt is where you either show up or vanish. Publishers with licensing leverage face a different calculus — but a merchant’s product pages benefit from every engine that can read them. The full reasoning is in our AI search optimization guide and the Shopify robots.txt guide.
One more thing robots.txt cannot do: make you citable. That takes structured data, an llms.txt manifest, and content engines can lift answers from — the layer RankEngine automates for Shopify stores.
Frequently asked questions
What does robots.txt actually do?
It tells crawlers which paths they may fetch. It does not remove pages from the index — a blocked URL can still be indexed from external links, just without content. To keep a page out of results, use a noindex meta tag and let crawlers see it. robots.txt is for managing crawl access, not visibility.
Should I block AI crawlers like GPTBot and ClaudeBot?
For an online store, usually no — blocking AI crawlers makes your products invisible to ChatGPT, Claude, and Perplexity exactly when shoppers are asking them what to buy. The nuanced move this generator supports: distinguish training crawlers (GPTBot, CCBot) from answer-engine crawlers (OAI-SearchBot, PerplexityBot) and decide each policy deliberately.
How do I edit robots.txt on Shopify?
Shopify serves an auto-generated robots.txt you cannot edit directly. It is customised through a robots.txt.liquid template in the theme, which keeps Shopify’s default rules (cart, checkout, admin) as long as the template still renders them — so only what you want to change is added. That template is theme code: hand the rules to your theme developer, or use an SEO app that manages it — RankEngine publishes its AI-crawler rules there for you.
What is the Content-Signal line?
An emerging convention (promoted by Cloudflare) that declares how content may be used: search=yes/no, ai-train=yes/no, ai-input=yes/no. Crawlers are beginning to respect it as a machine-readable usage policy alongside per-bot rules. This generator writes it to match the choices you make above.
Where does the file go?
At the root of the domain: yourstore.com/robots.txt. Search engines only check that exact path. On Shopify the robots.txt.liquid template handles placement automatically; on other platforms upload it to the web root.
All free tools
SEO Title Generator & Checker
Generate SEO titles from your keyword and check any title's character count, pixel width, and keyword position.
Meta Description Generator & SERP Preview
Generate high-converting meta descriptions and preview how your page looks in Google.
Keyword Density Checker
Paste any text and see keyword density, top 1–2–3-word phrases, and stuffing warnings.
LSI & Related Keyword Generator
Generate related keywords, question keywords, and commercial long-tails from any seed term.
Image Alt Text Generator & Checker
Build descriptive, SEO-friendly alt text for product images and check existing alt text for length, redundancy, and stuffing.
Googlebot Simulator — See What Google Sees
Fetch any URL with Googlebot's user agent: status code, redirects, title, meta robots, canonical, X-Robots-Tag, H1s, and an indexability verdict.
Schema markup generator
Generate valid JSON-LD schema free: Product, FAQPage, BreadcrumbList, LocalBusiness and Article.
Hreflang tag generator
Build a valid hreflang cluster for every language version of a page, with x-default, and a built-in check that every URL points back at every other one..
UTM builder
Build consistent UTM-tagged campaign URLs free.
URL slug generator
Turn any title into a clean, lowercase, hyphenated URL slug.
Readability checker
Paste text and get Flesch Reading Ease, Flesch-Kincaid grade level, sentence length distribution and the specific sentences that are dragging the score down..
SERP preview tool
Preview how a title, URL and meta description will render in Google on desktop and mobile, measured in pixels rather than characters so truncation is accurate..
SEO ROI calculator
Estimate what a ranking improvement is worth: search volume, position-based CTR, conversion rate and order value, plus a payback period on your SEO spend..
Open Graph generator
Generate Open Graph and Twitter card meta tags with a live preview of how the link will look when shared on Facebook, LinkedIn, X, WhatsApp and Slack..
SEO report generator
Generate an on-page SEO report for any URL: title and meta lengths, headings, canonical, schema types, image alt coverage, links and indexability, scored..
Redirect checker
Follow any URL hop by hop: every status code, every intermediate URL, and the page that finally answers.
Robots.txt tester
Fetch any site robots.txt and test a path against Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot.
Sitemap checker
Fetch any XML sitemap or index and check it: URL count against the 50,000 limit, lastmod coverage and honesty, size and reachability..
HTTP header checker
Fetch any URL as Googlebot and read the full response headers, with the SEO-relevant ones called out: X-Robots-Tag, Cache-Control, Content-Type, Vary and HSTS..
AI visibility checker
Check what AI answer engines may do with a site: GPTBot, ClaudeBot, PerplexityBot and Google-Extended rules, llms.txt, and whether pages are quotable..
hreflang checker
Read the hreflang cluster on any page: every declared alternate, whether the page includes itself, whether x-default is present, and whether the codes are valid..
Canonical checker
Check the canonical on any URL: present or missing, self-referencing or pointing elsewhere, and whether a redirect or noindex contradicts it..
Heading structure checker
Check the heading structure of any page: how many H1s, whether levels are skipped, and how many sections a reader or an answer engine can actually navigate..
Indexability checker
One verdict from every signal that decides it: status code, robots meta, X-Robots-Tag, canonical target, redirect chain and whether the HTML has content at all..
Related guides
ai.txt vs llms.txt: Which File Do AI Crawlers Read?llms.txt generator for Shopify (free): create oneHow to Get Indexed by LLMs Through an llms.txt FileShopify robots.txt: edit and fix it right
Run this across your whole catalog
RankEngine applies and verifies these fixes on every product in your Shopify store — automatically.
Start a 3-day free trial