AI crawler access
robots.txtMeta robotsX-Robots-Tag7 crawlers
Understand this checkScope, output, and practical use
What is it?
A deterministic test of whether known AI and search crawler user agents can access the submitted URL.
What does it check?
Reads robots.txt, page robots metadata, response headers, sitemap declarations, and path-specific rules.
Why is it needed?
A crawler cannot retrieve content that the site intentionally or accidentally blocks.
What does the output mean?
Allowed means the tested rules permit retrieval; blocked or partial points to the exact matching evidence. It does not guarantee indexing or citation.
Example use case
Use it after changing robots.txt to verify that OAI-SearchBot is allowed while private paths remain blocked.
No scan selected
Submit a public domain or page URL to inspect crawler access controls.