Let’s connect
SEO Library Technical SEO 1 min read

X-Robots-Tag

In short

The X-Robots-Tag is an HTTP response header that gives search engines indexing instructions — like noindex or nofollow — for any file type, including PDFs, images and other non-HTML resources.

Last reviewed · Iftikhar Qureshi

On this page

Why use a header instead of a meta tag?#

The meta robots tag only works inside HTML. PDFs, Word documents, images and feeds can’t contain it — the X-Robots-Tag header can. It’s also handy for applying rules to many files at once at the server level.

Examples#

Response header for a single file:

X-Robots-Tag: noindex, nofollow

Apache (.htaccess) — noindex all PDFs:

<FilesMatch ".pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Nginx:

location ~* .pdf$ {
  add_header X-Robots-Tag "noindex";
}

Target a specific bot: X-Robots-Tag: googlebot: noindex.

Tips#

  • It supports the same directives as meta robots (noindex, nofollow, nosnippet, max-image-preview…).
  • The file must be crawlable — don’t block it in robots.txt.
  • Check headers with DevTools (Network tab) or curl -I https://example.com/file.pdf.
  • Staging sites can send X-Robots-Tag: noindex sitewide as a safety net (plus password protection).

Related: noindex, robots.txt.

Share this term
SEO Library Technical SEO 1 min read

X-Robots-Tag

In short

The X-Robots-Tag is an HTTP response header that gives search engines indexing instructions — like noindex or nofollow — for any file type, including PDFs, images and other non-HTML resources.

Last reviewed · Iftikhar Qureshi

On this page

Why use a header instead of a meta tag?#

The meta robots tag only works inside HTML. PDFs, Word documents, images and feeds can’t contain it — the X-Robots-Tag header can. It’s also handy for applying rules to many files at once at the server level.

Examples#

Response header for a single file:

X-Robots-Tag: noindex, nofollow

Apache (.htaccess) — noindex all PDFs:

<FilesMatch ".pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

Nginx:

location ~* .pdf$ {
  add_header X-Robots-Tag "noindex";
}

Target a specific bot: X-Robots-Tag: googlebot: noindex.

Tips#

  • It supports the same directives as meta robots (noindex, nofollow, nosnippet, max-image-preview…).
  • The file must be crawlable — don’t block it in robots.txt.
  • Check headers with DevTools (Network tab) or curl -I https://example.com/file.pdf.
  • Staging sites can send X-Robots-Tag: noindex sitewide as a safety net (plus password protection).

Related: noindex, robots.txt.

Share this term