AI bots allowed in robots.txt
Your robots.txt is blocking AI assistants from reading your site. Open access for GPTBot, ClaudeBot, Google-Extended and PerplexityBot so ChatGPT and friends can find you.
What this means
Passing: Your robots.txt welcomes AI assistants. ChatGPT, Claude, Gemini and Perplexity can all read your site.
Partial: Your robots.txt blocks some AI assistants. Check the rules for GPTBot, ClaudeBot, Google-Extended and PerplexityBot.
Needs work: Your robots.txt is blocking AI assistants from reading your site. Open access for GPTBot, ClaudeBot, Google-Extended and PerplexityBot so ChatGPT and friends can find you.
Allow AI crawlers in /robots.txt
robots.txt is allow-by-default, so AI crawlers are blocked only by an explicit Disallow — you don't need to name each bot. Act only if a Disallow rule keeps crawlers from content you want surfaced in AI answers. Note: Google-Extended is separate — it controls whether Google may use your content to train and ground Gemini and Vertex AI, not whether Google can crawl or index your site.
Want the exact, copy-paste snippet for your own site? Run a free audit and sign in to see it in your report.
Full guide
Step-by-step for your platform, with why AI engines read it and how to confirm it worked: robots.txt: AI-bot allowlist + Sitemap.