Seotoolsdesk.com
Bulk up to 100 • Detect noindex/none via HTTP headers

X-Robots-Tag Checker

Paste URLs — we’ll fetch response headers, follow redirects, and flag risky directives like noindex / none.

Paste up to 100 URLs (one per line)
Tip: X-Robots-Tag: noindex blocks indexing at the HTTP level (often used for PDFs, staging, or API pages).

Results

URL X-Robots-Tag HTTP Hops Time Issues
Run a check to see results here.
“none” typically implies noindex + nofollow.

Quick interpretation

X-Robots-Tag is an HTTP header that controls indexing and snippets.

  • OK header looks safe (or intentionally absent)
  • Warning noindex/none/nofollow, multiple headers, hop limit
  • Error fetch errors, loops, HTTP 4xx/5xx
Indexing control

X-Robots-Tag Checker: find HTTP-level noindex

Unlike meta robots, X-Robots-Tag works on the HTTP response and can apply to non-HTML files like PDFs. This bulk checker follows redirects and shows what the final response tells crawlers.

Common problems

  • noindex / none blocks indexing for the final URL.
  • Redirect chains can hide the final header you actually ship.
  • Multiple headers can create confusing combined directives.

FAQ

What is X-Robots-Tag?

It’s an HTTP header that controls indexing behavior. Unlike meta robots, X-Robots-Tag applies at the response header level and works for any file type, including PDFs or images.

How is it different from meta robots?

Meta robots only affects HTML pages. X-Robots-Tag controls indexing before the crawler even parses content and can be applied via server config, CDN or proxy rules.

What does “noindex” mean in X-Robots-Tag?

It instructs crawlers not to index the URL. If found in redirect chains, the final response controls indexing.

What does “none” mean?

It combines noindex and nofollow. This prevents indexing and stops passing link equity.

Why detect X-Robots-Tag through redirects?

Sometimes headers are added only on the final destination. Redirect chains can expose unexpected noindex/noarchive rules deployed at CDN or origin-layer config.

What does “multiple headers detected” mean?

It means multiple X-Robots-Tag headers appear on the same response. Crawlers combine all directives, so they may unintentionally stack forbidden rules.

Should every page have X-Robots-Tag?

No. Missing header is normal. This tool focuses on accidental blocking via noindex/none and misconfigured CDN rules.

Does X-Robots-Tag override canonical tags?

Yes. A page with noindex won’t be indexed regardless of canonical declarations because directives in response headers take precedence at crawling level.