Seotoolsdesk.com
Fetch or paste robots.txt • Test up to 100 paths

Robots.txt Tester (Basic)

Load a robots.txt file (from a site or pasted text), pick a User-agent, and test if paths are allowed or blocked.

Robots source
If you leave User-agent empty, we test as *.
Paste robots.txt (optional)
Paste up to 100 URL paths to test (one per line)

Results

Path Decision Matched rule Matched group Notes
Run a test to see results here.
Rule selection: longest match wins; if tie — Allow wins.

How to use it

Test critical URLs: homepage, important categories, and “money pages”.

  • OK = Allowed
  • Warning = Disallowed (blocked)
  • Fetch errors are shown in the left panel notes.
Robots rules testing

Robots.txt Tester (Basic): verify blocks before Google does

Robots mistakes are silent. This tester helps you confirm which paths a crawler can access for a given user-agent.

What this basic tester supports

  • User-agent groups
  • Allow / Disallow matching (with * wildcard, optional $ end anchor)
  • Longest-match rule resolution (Allow wins ties)

FAQ

What is a robots.txt tester?

It simulates how a crawler evaluates Allow/Disallow rules for a given User-agent. You paste (or fetch) robots.txt, then test up to 100 paths to see if they’re allowed or blocked.

Fetch vs paste: what’s the difference?

Fetch downloads /robots.txt from the site (with redirects). Paste lets you test rules before publishing them — perfect for staging or drafts.

Can I paste full URLs in the test list?

Yes. If you paste a full URL, the tool extracts the path (and query) and tests that value.

How does it choose which user-agent group applies?

The tool selects groups where the group’s user-agent matches your input user-agent. More specific matches win (longer match). If nothing matches, it falls back to * when available.

Which rule wins: Allow or Disallow?

It uses common behavior: longest matching pattern wins. If two rules tie on match length, Allow wins.

Do you support wildcards and the $ end anchor?

Basic support: * works as a wildcard inside patterns, and $ at the end is treated as an end-of-line anchor.

If robots.txt is empty, what happens?

With no rules, everything is treated as Allowed by this tester. (Real crawlers generally treat an empty file as no restrictions, too.)

Why do I see errors but still get results?

Fetch warnings (404, empty file, HTTP errors) are shown as notes, but the tester still tries to parse what it has so you can see behavior and matched rules.

Is this the same as Google’s robots testing?

No. This is a fast “basic” tester for common rules and conflict resolution. Different bots may interpret edge cases differently. Use it to catch obvious blocks and verify rule intent.