Seotoolsdesk.com
Bulk up to 100 • meta robots + X-Robots-Tag

Robots Meta Tag Checker

Paste URLs — we’ll detect indexing directives from meta name="robots", meta name="googlebot", and the X-Robots-Tag header.

Paste up to 100 URLs (one per line)
Tip: noindex can come from meta tags or from the X-Robots-Tag header.

Results

URL Directives HTTP Time Issues
Run a check to see results here.
“Blocked” means we detected noindex (meta or header). Fix only if the page should be indexable.

How to read the output

Indexing signals can come from multiple places.

  • OK no blocking directives found
  • Warning no directives or nofollow present
  • Error noindex detected or HTTP/fetch errors
Indexing directives

Robots Meta Tag Checker: catch accidental noindex

A single noindex directive can remove a page from search results — even if everything else is perfect. This checker helps you quickly find blocks from meta tags and server headers across a batch of URLs.

What is checked

  • Meta robots (name="robots")
  • Meta googlebot (name="googlebot")
  • X-Robots-Tag header (server/CDN-level blocks)

FAQ

What does this checker scan?

It checks three sources of robots directives on the final URL (after redirects): meta name="robots", meta name="googlebot", and the X-Robots-Tag HTTP response header.

Why is “Blocked” shown?

“Blocked” means we detected noindex (or none) in any place — meta tags or X-Robots-Tag. If a page should rank, remove the blocking directive.

What is X-Robots-Tag and why is it important?

X-Robots-Tag is an HTTP header that can set indexing rules even for non-HTML files (PDFs, images) and can be applied at server or CDN level. It can override your expectations if you only check HTML.

Do you follow redirects?

Yes. The tool follows redirects up to a safe limit and then evaluates robots directives on the final destination. Redirect loops are detected and flagged.

Should every page have a meta robots tag?

No. If there is no robots directive, most crawlers assume the default behavior (index,follow). Many sites leave it empty intentionally. This tool still flags “no directives” as a warning for audit clarity.

What’s the difference between “noindex” and “nofollow”?

noindex prevents a page from being indexed and shown in search. nofollow tells crawlers not to follow links from that page (or reduces link-following). none typically means noindex,nofollow.

Why do I see HTTP 0?

HTTP 0 means the request did not complete (no valid HTTP response). Common causes include DNS failure, TLS/SSL errors, firewall/CDN blocks, or timeouts.

Can multiple X-Robots-Tag headers exist?

Yes. Some servers return multiple X-Robots-Tag lines (or different values for different user-agents). This tool collects them and merges them into a single combined view.

Is a robots.txt rule checked here?

No. This tool checks page-level directives (meta + headers). robots.txt is a separate mechanism and should be checked with a Robots.txt Checker / Tester.