Noindex
Noindex tells engines to keep a page out of the index. Learn how it differs from robots.txt disallow and why combining the two backfires.
TL;DR
Noindex is a directive telling search engines to keep a page out of their index, delivered either as a meta robots tag or an X-Robots-Tag header. The critical detail is that a crawler has to fetch the page to read the directive. Blocking the URL and adding noindex are mutually exclusive strategies, and combining them produces the opposite of what people expect.
Why noindex matters
Most sites generate pages that need to exist for users and should never appear in search results, and noindex is the correct tool for exactly that situation.
- Thin page control: Tag archives, filtered views and paginated duplicates can stay usable without diluting your indexed footprint.
- Private but public pages: Thank-you pages, internal search results and staging URLs stay reachable and unindexed.
- Quality signals: A smaller index of substantive pages reads better than a large one padded with near-duplicates.
- Link flow preserved: Unlike a robots.txt block, a noindexed page is still crawled, so links on it are still followed.
How to implement noindex
Choose your delivery method
A meta robots tag for HTML pages, an X-Robots-Tag header for PDFs, images and other non-HTML files.
Ensure the page is crawlable
The URL must not be disallowed in robots.txt, or the directive is never read.
Decide about following links
Noindex alone still follows links. Adding nofollow stops link equity flowing from the page, which is usually not what you want.
Remove the URL from your sitemap
Listing a noindexed page tells engines to index something you have asked them not to index.
Verify in the raw response
Check the meta tag appears in the HTML source, not just the rendered DOM, since script-injected directives are frequently missed.
Audit after releases
A staging noindex shipped to production is one of the most common and most damaging deployment errors in web publishing.
Noindex vs. robots.txt disallow
Noindex: The page is fetched, the directive is read, and the URL is dropped from the index. Links on the page are still followed.
Robots.txt disallow: The page is never fetched. Nothing on it is read, including a noindex tag, and the URL can still appear in results if other sites link to it.
Using both on the same URL is the classic mistake. The disallow prevents the fetch, so the noindex is never seen, and the page can linger in results as a bare URL indefinitely. Pick one: disallow to save crawl budget, noindex to remove from the index.
A noindex left on a production template removes pages from search and AI answers with no error anywhere in your monitoring. Findrix checks index directives, crawler access and rendering as part of a 31-point technical audit. Every gap comes with the fix already written: technical, content and off-site. The audit is free, takes about a minute, and requires no signup.
Pages that usually deserve noindex
- Internal search results: Infinite low-value URL combinations that add nothing to an index.
- Thank-you and confirmation pages: Necessary for users, meaningless in search results.
- Faceted filter combinations: Beyond the few that attract genuine search demand.
- Thin tag archives: Category pages listing two posts and no original content.
- Staging and development environments: Where a robots.txt block plus authentication is usually the better answer.
The deployment error that costs quarters
Staging environments are noindexed for good reason. The trouble comes when the noindex ships to production alongside a release, because nothing about the site looks broken afterwards: pages load, tests pass, analytics reports traffic from existing visitors.
Search traffic then decays over several weeks as engines re-crawl and drop pages, which is slow enough that teams often attribute it to seasonality or an algorithm update before checking the tag.
Add an automated post-deploy check that fetches a few production URLs and fails loudly if a noindex directive appears.
Frequently asked questions
What is the difference between noindex and nofollow?
Noindex keeps a page out of the index. Nofollow tells engines not to pass authority through links. They are independent, and using noindex alone is usually correct, since you generally still want links on the page to be followed.
How long does noindex take to work?
Engines have to re-crawl the page to see the directive, which can take days for frequently crawled URLs and weeks for deep ones. Requesting removal in Search Console speeds up the process for urgent cases.
Does noindex affect AI answers?
Indirectly. Retrieval-based engines lean heavily on indexed content, so a noindexed page is far less likely to be surfaced or cited. If a page matters for AI visibility, it should be indexable.