To diagnose missing crawler access, inspect both the delivered crawl preferences and the actual page response. A crawler can be allowed by robots.txt and still be blocked by a login, firewall rule, challenge, or failed response.
Identify the request you want to allow#
Decide whether you are evaluating search indexing, model-training crawls, or user-triggered retrieval. Providers can use different agents for these purposes. Use their current published crawler documentation when selecting agent names or verifying identities.
User-agent strings can be spoofed. A name in request analytics is a classification of observed traffic unless additional verification supports the identity. Likewise, the absence of a recognizable name does not prove an assistant never used your content through another retrieval path.
Inspect the public response#
- Open
https://your-host/robots.txton the exact live hostname. - Read the relevant named user-agent group and wildcard group together.
- Request an important page and follow redirects to its final URL.
- Confirm a successful page response with useful content in the HTML.
- Check firewall, bot-management, and authentication controls if the request is challenged or rejected.
A rule such as the following concerns only the named group:
User-agent: ExampleBot
Disallow: /
Allow: /docs/
ExampleBot is a placeholder. This example does not block every crawler, authenticate private content, or guarantee that a permitted page will be cited.
Use the right publishing path#
For an Answer Cloud-hosted website, discovery files are generated by the platform. Check the intended source and deployment, then report an unexpected generated rule through support. Do not ship a competing robots implementation to work around a platform-generated file.
For a website that stays on another host, manage that website's robots and security controls there. If Cloudflare proxies the site, inspect managed robots settings separately from bot-blocking controls. Historical screenshots are not a current configuration guide; verify the actual delivered file after a change.
Confirm the result#
Repeat the original failing page request and compare its status and content. Check whether relevant observed requests begin arriving, allowing for provider timing and reporting delay. A test of your own access establishes reachability for that test, not successful retrieval by every provider.
Keep private information behind real access controls. robots.txt expresses preferences to cooperating crawlers; llms.txt guides content discovery. Neither protects secrets or replaces authentication.
Continue with generated discovery resources and bot classifications.