Four signals govern whether a search crawler can index a page. One tool reconciles them into a single verdict.

robots.txt directives, meta robots tags, X-Robots-Tag headers, and canonical links can contradict each other. Most tools report each signal separately. This one states the verdict and names the rule that produced it.

Check a URL

How it works

  1. 01 Paste any URL into the input field
  2. 02 Robotsread fetches the page and its robots.txt
  3. 03 All four indexability signals are parsed and reconciled
  4. 04 A single verdict is returned—blocked, allowed, or uncertain—with the specific rule cited

The four signals

robots.txt

Disallow and Allow directives that match the requested path. A matching Disallow blocks crawling; a matching Allow overrides it for the same path. Only <Googlebot> and <*> user-agent groups are evaluated.

<meta name="robots">

A noindex directive in the page HTML overrides every other signal. This tag tells crawlers not to include the page in search results, regardless of robots.txt permissions.

X-Robots-Tag

An HTTP response header that can set noindex for non-HTML resources (PDFs, images) or override meta tags. It carries the same authority as a meta robots tag but lives outside the document body.

rel="canonical"

A link element pointing to a different URL. This does not block indexing by itself, but signals that the canonical page—not this one—should appear in search results. The verdict is marked as uncertain.

Limitations

This tool reports what the rules say, not what any search engine will actually do. It does not test rendering, JavaScript execution, or ranking eligibility. A host behind Cloudflare cannot be reached from this platform—those checks return an explicit platform limitation message, never a fabricated finding about the site.

No email, no alerts, no scheduled re-checks. One URL, one verdict, stated plainly.

Check a URL now