Check if AI crawlers can access your website

Check AI crawler access from any page

Most crawler checks only look at a domain. The checker reads the robots.txt published at the exact origin you are on — scheme, host and port — then evaluates the rules against the page path you are actually viewing, including its query string. A site can allow crawling at the root and still block a template, a section or a single document, so a domain-level answer is often wrong for the page that matters.

Matching follows the Robots Exclusion Protocol: specific user-agent groups take precedence over the wildcard group, * and $ patterns are honoured, and the most specific matching rule wins, with Allow preferred when rules are equally specific.

Supported crawler controls

What the checker can and cannot tell you

Results use four precise statuses: Allowed by robots.txt, Blocked by robots.txt, No robots.txt restriction detected and Unable to verify. There is no score and no percentage, because robots.txt does not support one.

Why crawler access matters

Crawler access is a technical prerequisite. If a search or model crawler is blocked from a page, that page cannot be retrieved through that route at all — so any downstream outcome is off the table before content, structure or authority is considered. Blocking is often unintentional: rules copied between environments, a staging directive shipped to production, or a broad Disallow written before AI crawler tokens existed.

Checking access tells you the door is open. It does not tell you anyone walked through it.