# Manage crawler access and diagnose blocked content | answer.cloud Docs

> Canonical source: [Manage crawler access and diagnose blocked content | answer.cloud Docs](https://www.answer.cloud/docs/ai-visibility/crawler-access/)

Check crawler preferences, public page access, and CDN controls while keeping robots rules, indexing choices, and private access separate.

guide

Manage crawler access and diagnose blocked content

Check crawler preferences, public page access, and CDN controls while keeping robots rules, indexing choices, and private access separate.

Reviewed November 1, 2026

To diagnose missing crawler access, inspect both the delivered crawl preferences and the actual page response. A crawler can be allowed by robots.txt and still be blocked by a login, firewall rule, challenge, or failed response.

## Identify the request you want to allow

Decide whether you are evaluating search indexing, model-training crawls, or user-triggered retrieval. Providers can use different agents for these purposes. Use their current published crawler documentation when selecting agent names or verifying identities.

User-agent strings can be spoofed. A name in request analytics is a classification of observed traffic unless additional verification supports the identity. Likewise, the absence of a recognizable name does not prove an assistant never used your content through another retrieval path.

## Inspect the public response

1. Open `https://your-host/robots.txt` on the exact live hostname.
2. Read the relevant named user-agent group and wildcard group together.
3. Request an important page and follow redirects to its final URL.
4. Confirm a successful page response with useful content in the HTML.
5. Check firewall, bot-management, and authentication controls if the request is challenged or rejected.

A rule such as the following concerns only the named group:

```
User-agent: ExampleBot
Disallow: /
Allow: /docs/
```

`ExampleBot` is a placeholder. This example does not block every crawler, authenticate private content, or guarantee that a permitted page will be cited.

## Use the right publishing path

For an Answer Cloud-hosted website, discovery files are generated by the platform. Check the intended source and deployment, then report an unexpected generated rule through [support](https://www.answer.cloud/docs/account/support/). Do not ship a competing robots implementation to work around a platform-generated file.

For a website that stays on another host, manage that website's robots and security controls there. If Cloudflare proxies the site, inspect managed robots settings separately from bot-blocking controls. Historical screenshots are not a current configuration guide; verify the actual delivered file after a change.

## Confirm the result

Repeat the original failing page request and compare its status and content. Check whether relevant observed requests begin arriving, allowing for provider timing and reporting delay. A test of your own access establishes reachability for that test, not successful retrieval by every provider.

Keep private information behind real access controls. `robots.txt` expresses preferences to cooperating crawlers; `llms.txt` guides content discovery. Neither protects secrets or replaces authentication.

Continue with [generated discovery resources](https://www.answer.cloud/docs/ai-visibility/discovery-files/) and [bot classifications](https://www.answer.cloud/docs/analytics/bots/).