# AI Crawling Preferences: robots.txt vs llms.txt

> Canonical source: [AI Crawling Preferences: robots.txt vs llms.txt](https://www.answer.cloud/pages/ai-crawling-permissions-robots-txt-vs-llms-txt-hosted/)

Use robots.txt for crawler preferences and llms.txt for content guidance, with authentication for private information.

`robots.txt` and `llms.txt` serve different purposes. Neither should be treated as an authentication mechanism.

## robots.txt: crawl preferences

Cooperating crawlers use robots.txt rules to decide which URLs they may crawl. A rule applies to the relevant user-agent group; `Disallow: /` for one named crawler does not necessarily block every crawler.

For a self-managed site, a rule allowing a named crawler only under docs can use:

```text
User-agent: ExampleBot
Disallow: /
Allow: /docs/

Sitemap: https://example.com/sitemap.xml
```

`ExampleBot` is a placeholder. Select actual provider agents and purposes deliberately: training, search indexing, and user-triggered retrieval can use different agents. Consult the provider’s current crawler documentation.

Robots rules do not protect private pages or necessarily remove known URLs from a search index. Use real access controls and appropriate indexing mechanisms for those goals.

## llms.txt: a content guide

The [llms.txt proposal](https://llmstxt.org/) describes a Markdown overview with named links to useful resources. It does not use Allow/Disallow directives or guarantee that a provider will retrieve or cite the listed content.

## answer.cloud workflow

The platform generates discovery files during deployment and CMS publication. Keep content accurate, review access to important pages, and verify results. See the [llms.txt guide](https://www.answer.cloud/pages/llms-txt-implementation-guide-templates-validation-and-deployment-hosted/) and [robots.txt guide](https://www.answer.cloud/pages/how-to-enable-chatbots-to-crawl-your-website-hosted/).