robots.txt and llms.txt serve different purposes. Neither should be treated as an authentication mechanism.
robots.txt: crawl preferences
Cooperating crawlers use robots.txt rules to decide which URLs they may crawl. A rule applies to the relevant user-agent group; Disallow: / for one named crawler does not necessarily block every crawler.
For a self-managed site, a rule allowing a named crawler only under docs can use:
User-agent: ExampleBot
Disallow: /
Allow: /docs/
Sitemap: https://example.com/sitemap.xml
ExampleBot is a placeholder. Select actual provider agents and purposes deliberately: training, search indexing, and user-triggered retrieval can use different agents. Consult the provider’s current crawler documentation.
Robots rules do not protect private pages or necessarily remove known URLs from a search index. Use real access controls and appropriate indexing mechanisms for those goals.
llms.txt: a content guide
The llms.txt proposal describes a Markdown overview with named links to useful resources. It does not use Allow/Disallow directives or guarantee that a provider will retrieve or cite the listed content.
answer.cloud workflow
The platform generates discovery files during deployment and CMS publication. Keep content accurate, review access to important pages, and verify results. See the llms.txt guide and robots.txt guide.