Knowledge base

robots.txt: AI-bot allowlist + Sitemap

Medium impactCrawl & access

robots.txt is a small text file at the root of your site that tells visiting bots what they may read; this fix rewrites it to explicitly welcome the AI crawlers and point them to your sitemap.

Why AI engines read this

If that file accidentally blocks AI crawlers, they never read your site at all, so welcoming them is the basic permission slip that lets you show up in AI answers.

How to tell it applies to you

robots.txt is missing, a Disallow blocks an AI crawler, or there's no Sitemap: directive.

Apply it on your platform

The exact place to make this change depends on your platform. Open the guide for the step-by-step on WordPress, Shopify, Wix, and more.

Open the platform guide

Step-by-step for your CMS, with how to confirm it worked.

Continue →

How you'll know it's done

Mark the fix applied in your dashboard once you've made the change — Findrix re-checks the live page right away, and the weekly scan confirms it. Once the fix is live the matching check flips to pass; a change you made without marking it is caught by the same scan.