Robots.txt
Robots.txt tells crawlers which parts of your site they may fetch. Learn the syntax, the AI crawler rules, and the mistake that hides you from AI answers.
TL;DR
Robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may and may not fetch. It works by user agent, so different bots can be given different rules in the same file. It is the single most consequential file for AI visibility, because a crawler blocked here never reads your pages and an engine that never reads your pages cannot cite them.
Why robots.txt matters
Every decision in this file is a decision about who gets to know what your site says.
- AI answer eligibility: Blocking a retrieval crawler removes you from that engine answers entirely, with no warning and no error anywhere.
- Crawl efficiency: Excluding search, filter and session URLs keeps crawlers on pages that matter.
- Server load: Aggressive bots consume real bandwidth, and rules are the cheapest way to limit them.
- Training control: Training crawlers and retrieval crawlers can be treated differently, which is the distinction most sites get wrong.
How to configure robots.txt
Place it at the root
Only yourdomain.com/robots.txt is read. A file in a subdirectory is ignored.
Group rules by user agent
Each User-agent line starts a block, followed by the Allow and Disallow rules that apply to it.
Separate training from retrieval crawlers
GPTBot and Google-Extended feed training. OAI-SearchBot and PerplexityBot feed live answers. Blocking the second group costs you citations.
Block low-value paths, not sections
Internal search results, faceted URL parameters, cart and account pages. Be precise, since a broad rule catches more than intended.
Reference your sitemap
A Sitemap line pointing at your XML sitemap helps crawlers find your URL inventory.
Test before and after shipping
Use a robots.txt tester on your most important URLs, and re-test after any change to the file.
Robots.txt vs. noindex
Robots.txt: Controls fetching. A disallowed URL is never requested, so the crawler never sees what is on it, including any instructions.
Noindex: Controls indexing. The crawler fetches the page, reads the directive, and leaves it out of the index.
The trap is using both together. A page disallowed in robots.txt is never fetched, so its noindex tag is never read, and the URL can persist in results indefinitely. Choose one mechanism per outcome.
A single misplaced Disallow can remove you from an engine answers without producing an error anywhere you would look. Findrix checks crawler access for every major AI bot as part of a 31-point technical audit, then tracks how often seven engines cite you against named competitors. Every gap comes with the fix already written: technical, content and off-site. The audit is free, takes about a minute, and requires no signup.
AI crawlers worth knowing by name
- GPTBot: OpenAI training crawler. Blocking it limits use of your content in model training.
- OAI-SearchBot: OpenAI retrieval crawler for ChatGPT search. Blocking it removes you from those answers.
- ClaudeBot: The Anthropic crawler.
- PerplexityBot: The Perplexity crawler, which feeds a heavily citation-based product.
- Google-Extended: Controls Gemini training use without affecting Google Search indexing.
- Applebot-Extended: The Apple equivalent training control.
The blanket block that quietly costs citations
The common sequence goes like this. Somebody reads that AI companies train on web content without paying for it, decides that is unacceptable, and adds a broad set of Disallow rules covering every AI user agent they can find.
The intent is to prevent training. The effect includes removing the site from AI answers, because retrieval crawlers were caught in the same net. Months later the brand is absent from ChatGPT and Perplexity responses in its own category, and nothing in analytics explains why.
To opt out of training while staying citable: block GPTBot, Google-Extended and Applebot-Extended, and allow OAI-SearchBot and PerplexityBot.
Frequently asked questions
Does robots.txt remove a page from Google?
No. It stops the page being fetched, but a URL with external links can still appear in results without a description. Use a noindex directive on a fetchable page to remove it from the index.
Should I block AI crawlers in robots.txt?
Decide per bot rather than as a category. Blocking training crawlers limits use of your content in model training. Blocking retrieval crawlers removes you from AI answers, which is usually the opposite of what a marketing team wants.
Is robots.txt legally binding?
No. It is a convention that well-behaved crawlers respect voluntarily. Major operators honour it; bad actors ignore it. Genuine access control requires authentication rather than a text file.