X-Robots-Tag
X-Robots-Tag delivers indexing directives in the HTTP header, so it works on PDFs and images. Learn the syntax and when to use it over a meta tag.
TL;DR
X-Robots-Tag is an HTTP response header that carries indexing directives, doing the same job as a meta robots tag without requiring the file to be HTML. Because it lives in the header rather than the document, it works on any file type a server can return. That difference is the whole reason it exists: a PDF has no head section to put a meta tag in, and PDFs are indexed constantly.
Why X-Robots-Tag matters
Non-HTML files accumulate quietly on most sites and end up in search results without anyone deciding they should be there.
- Non-HTML coverage: PDFs, spreadsheets, images and videos can be noindexed only through the header.
- Pattern-based rules: Server configuration can apply directives to whole file types or directories at once.
- No template changes: Indexing rules ship through server config rather than requiring edits to every document.
- Additional directives: Options like noarchive, nosnippet and max-snippet are available alongside noindex.
How to implement X-Robots-Tag
Choose the right layer
Apache uses .htaccess or the main config, nginx uses add_header, and application frameworks can set it per response.
Target by file type or path
Match on extension for blanket rules on PDFs, or on directory for a section that should stay unindexed.
Name the user agent when needed
Directives can be scoped to a specific crawler, allowing different rules for different engines.
Combine directives with commas
Multiple instructions can travel in a single header value.
Confirm the page is crawlable
As with meta noindex, a URL disallowed in robots.txt is never fetched, so the header is never read.
Verify with a header check
Fetch the URL and inspect the response headers directly rather than assuming the config applied.
X-Robots-Tag vs. meta robots tag
X-Robots-Tag: Delivered in the HTTP header. Works for any file type, configurable in bulk at the server level, invisible in the page source.
Meta robots tag: Delivered in the HTML head. Works only for HTML pages, set per page, visible to anyone viewing source.
They support the same directives and engines treat them equivalently. Use the meta tag for HTML pages, where it is easier to see and debug, and the header for everything else.
Header-level directives are easy to apply too broadly and hard to notice afterward, since nothing appears in the page source. Findrix checks index directives across page types as part of a 31-point technical audit. Every gap comes with the fix already written: technical, content and off-site. The audit is free, takes about a minute, and requires no signup.
Directives X-Robots-Tag supports
- noindex: Keep this resource out of the index.
- nofollow: Do not follow links found in this resource.
- noarchive: Do not show a cached copy.
- nosnippet: Show no text snippet or video preview in results.
- max-snippet: Cap the snippet length at a set number of characters.
- unavailable_after: Drop the resource from the index after a stated date.
Where it usually gets applied wrongly
The characteristic failure is scope. A rule written to noindex one report directory matches a broader path than intended, and a section of the site quietly leaves the index with no visible change to any page.
Because the directive lives in the header, nobody finds it by viewing source. It surfaces only in a header inspection or a crawl tool configured to report response headers, neither of which is part of a normal content review.
When a meta tag and a header disagree, engines take the more restrictive instruction. Check both when a page is missing for no apparent reason.
Frequently asked questions
When should I use X-Robots-Tag instead of a meta tag?
For anything that is not an HTML page: PDFs, images, videos, spreadsheets. Also when you need to apply a rule across many URLs at once, since server config handles patterns better than template edits.
Can X-Robots-Tag block AI crawlers?
It can carry directives scoped to specific user agents, so it can express rules for named AI crawlers. Robots.txt remains the conventional place for crawl-level access control, with X-Robots-Tag better suited to indexing behaviour.
How do I check whether a page has an X-Robots-Tag?
Fetch the URL and read the response headers, using curl, your browser network panel, or a crawler that reports headers. It never appears in the page source, which is why it is so often overlooked.