# Review Cloudflare Managed robots.txt Rules

> Canonical source: [Review Cloudflare Managed robots.txt Rules](https://www.answer.cloud/pages/a-cloudflare-managed-robots-txt-warning-hosted/)

Check the delivered robots.txt and bot controls separately; named crawler blocks do not mean every crawler is blocked.

If Cloudflare proxies your website, review the robots.txt response actually delivered to visitors and crawlers. Also inspect the separate bot-management controls. Their purposes and effects differ.

## Read the matching user-agent group

The historical screenshots below allow the wildcard group while disallowing selected named AI crawlers. They do not show a blanket block on all bots. `Disallow: /` applies to the relevant matching group; it must be interpreted with the rest of the file.

## Use current controls

Follow Cloudflare’s [managed robots.txt documentation](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) for the current setting and behavior. The screenshots are a January 2026 example, not a current click-by-click setup guide.

Verify the public response after changes and test relevant pages. Robots preferences are separate from security rules that may block a request, and neither a permissive robots file nor disabling a setting guarantees AI visibility.

Read [robots.txt vs llms.txt](https://www.answer.cloud/pages/ai-crawling-permissions-robots-txt-vs-llms-txt-hosted/).


![Article illustration](https://www.answer.cloud/assets/cms/article-23702-551364bb8887.png)


![Article illustration](https://www.answer.cloud/assets/cms/article-23702-7a9830c94c84.jpg)