Use robots.txt to express crawl preferences to cooperating crawlers. Choose which provider agents and purposes to allow; search discovery, model training, and user-triggered retrieval are separate activities.
A self-managed website example
User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xml
Replace example.com with the actual site and review more specific user-agent groups. This example requests that cooperating crawlers avoid private paths; it does not secure those paths. Protect sensitive content with authentication.
Do not add an undocumented LLMmap directive or assume that an llms.txt comment creates a crawl rule. Robots preferences, sitemaps, and a Markdown content guide have different roles.
Verify the result
Check the public robots.txt response, relevant provider rules, and actual access to your pages. A firewall, login requirement, or failed response can still prevent retrieval even when a crawl rule allows it. Allowing a crawler does not guarantee indexing or a citation.
On answer.cloud-hosted sites, discovery files are generated by the platform. Use the supported publishing workflow rather than shipping manual replacements. Read robots.txt vs llms.txt.