Publish a root file (robots.txt / sitemap.xml / llms.txt)
A root file is a plain-text or XML file such as robots.txt, sitemap.xml, or llms.txt. It is served from a fixed address at your site's web root so search crawlers and AI models find it at a predictable URL. robots.txt controls which crawlers may access your pages, sitemap.xml lists every page you want indexed, and llms.txt is a curated list of your most valuable pages written for AI models. Each website platform exposes its own slot for placing or editing these files.
Search crawlers and AI models look for these files at fixed root URLs before they read the rest of your site, so a correct robots.txt, sitemap.xml, and llms.txt decides what gets crawled, how completely your pages are discovered, and which pages AI assistants treat as your canonical content. Without them, crawlers guess at your structure, miss pages that aren't linked from your homepage, and have no curated shortlist to cite. Publishing them yourself puts you in control of what machines find and how they represent your business.
Do it on your platform
Pick your website platform below for the exact place to make this change, and how to confirm it worked.
- 1
In the WordPress dashboard, open Plugins → Add New if you don't already run an SEO plugin, then install and activate Yoast SEO or Rank Math. All three files run through one of these plugins.
PluginsAdd New - 2
For sitemap.xml, do nothing: both Yoast SEO and Rank Math generate the sitemap automatically once active and serve it at /sitemap_index.xml. Note it is NOT at /sitemap.xml.
- 3
For robots.txt in Yoast SEO, open Yoast SEO → Tools → File editor, paste your rules into the robots.txt box, and click 'Save changes to robots.txt'. WordPress serves the result at /robots.txt.
Yoast SEOToolsFile editor - 4
For robots.txt in Rank Math instead, open Rank Math → General Settings → Edit robots.txt and enter your rules there.
Rank MathGeneral SettingsEdit robots.txt - 5
For llms.txt in Yoast SEO, open Settings → Site features, find the 'AI tools' section, switch the 'LLMs.txt' toggle on, and click Save. Yoast generates and serves the file at /llms.txt.
Yoast SEOSettingsSite features - 6
For llms.txt in Rank Math instead, open Rank Math SEO → Dashboard, find the 'LLMS Txt' module, and switch it on.
Rank Math SEODashboard
Check it worked. Open yourdomain.com/robots.txt, yourdomain.com/sitemap_index.xml, and yourdomain.com/llms.txt in a browser. Each shows your content, and the sitemap shows a list of links rather than an error page.
Heads up. The Yoast 'File editor' tab appears only if your hosting lets WordPress write files; if it is missing, edit robots.txt through your host's File Manager or FTP instead. Native llms.txt support arrived in Yoast and Rank Math only in mid-2025, and Yoast relocated the toggle in 2026 (it previously sat under Advanced). Update the plugin and search 'LLMs.txt' if you cannot find it. Do not expect the sitemap at /sitemap.xml; the real address is /sitemap_index.xml.