Help Centre

What exactly does robotsread check?

It fetches the URL you provide and the /robots.txt file from the same origin. It then parses four indexability signals: robots.txt Disallow/Allow directives (for Googlebot and wildcard user-agents), <meta name="robots"> tags in the page HTML, X-Robots-Tag HTTP response headers, and rel="canonical" link elements. It reconciles any contradictions and returns one verdict — blocked, allowed, or uncertain — with the specific rule that produced it.

My page is blocked. What do I do next?

The verdict tells you which signal is blocking your page. The fix depends on where the block lives:

• robots.txt blocking: Edit your robots.txt file. Remove or modify the Disallow directive that matches your page's path. Remember that search engines cache robots.txt, so the change may take time to propagate.
• Meta robots noindex: Remove the <meta name="robots" content="noindex"> tag from the page HTML (or change it to index). This tag is authoritative — it overrides robots.txt permissions.
• X-Robots-Tag noindex: Remove the X-Robots-Tag: noindex HTTP header from your server configuration. This is common for non-HTML resources like PDFs.
• Canonical pointing elsewhere: If the canonical link points to a different URL, search engines may prefer that URL. Update the rel="canonical" to point to the page itself, or accept that the canonical URL is the intended index target.

After making changes, re-run the check to confirm the block is resolved.

Why does the tool say "uncertain"?

The verdict is marked as uncertain in a few situations:

• The page sets a rel="canonical" pointing to a different URL. This does not block indexing, but signals that the canonical URL — not this one — should appear in search results. The tool reports this as uncertain because the page could be indexed, but not as the primary result.
• The page returned a non-2xx HTTP status (e.g., 404, 500, 302 redirect). In this case, the page-body signals (meta robots, canonical) are parsed from the error page, not from the intended content, so the verdict has limited reliability.
• One or more signals could not be fetched or parsed. If the page could not be reached at all, the verdict is marked as uncertain rather than claiming a result from incomplete data.

Can robotsread check my entire site?

No. robotsread checks exactly one URL per request. It fetches only the URL you provide and that origin’s /robots.txt. It never crawls the site, follows links, or enumerates pages. There is no batch-checking feature available. If you need to check many pages, each must be submitted individually.

My site is behind Cloudflare and the tool says it cannot reach it. Is my site broken?

No. This is a platform limitation, not a finding about your site. Cloudflare blocks requests from the platform robotsread runs on. The tool displays this limitation as a clearly labelled message — it does not attempt to produce a verdict from an empty or error response. Your site is almost certainly fine; use another tool (such as Google’s URL Inspection tool in Search Console) to check indexability from a network that can reach Cloudflare hosts.

Does the tool tell me whether my page will actually rank?

No. robotsread reports what the indexability rules say — it does not speak for any search engine’s actual ranking or indexing behaviour. A page reported as "allowed" may still not appear in search results for many reasons: poor content quality, lack of backlinks, technical SEO issues the tool does not detect (JavaScript rendering, page speed, structured data errors), or simply because the search engine has not crawled it yet. This tool answers one question: “Is this page allowed to be indexed, and which rule governs that?”