Maintained resource

AI crawler access is a policy decision

robots.txt controls retrieval for compliant crawlers. It does not grant rights, guarantee inclusion, or make a page citation-worthy.

Separate crawler purposes

Search discovery, user-requested retrieval, and model training are different uses. Review the crawler owner’s current documentation and record why each bot is allowed or blocked.

Start with public-path intent

User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

This is an illustrative baseline, not a mandatory policy. Protect private paths with authentication and authorization, not robots.txt.

Verify all layers

Confirm robots.txt status, content type, and matching group.
Inspect meta robots and X-Robots-Tag headers.
Test the root and an important content path.
Confirm sitemap URLs are canonical, indexable, and successful.
Rerun after CDN, WAF, migration, or framework changes.