# How to Configure robots.txt for Chatbots and AI Crawlers

> Canonical source: [How to Configure robots.txt for Chatbots and AI Crawlers](https://www.answer.cloud/pages/how-to-enable-chatbots-to-crawl-your-website-hosted/)

Review crawler-specific preferences and verify page access without treating robots.txt as security or a citation guarantee.

Use robots.txt to express crawl preferences to cooperating crawlers. Choose which provider agents and purposes to allow; search discovery, model training, and user-triggered retrieval are separate activities.

## A self-managed website example

```text
User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
```

Replace example.com with the actual site and review more specific user-agent groups. This example requests that cooperating crawlers avoid private paths; it does not secure those paths. Protect sensitive content with authentication.

Do not add an undocumented `LLMmap` directive or assume that an llms.txt comment creates a crawl rule. Robots preferences, sitemaps, and a Markdown content guide have different roles.

## Verify the result

Check the public robots.txt response, relevant provider rules, and actual access to your pages. A firewall, login requirement, or failed response can still prevent retrieval even when a crawl rule allows it. Allowing a crawler does not guarantee indexing or a citation.

On answer.cloud-hosted sites, discovery files are generated by the platform. Use the supported publishing workflow rather than shipping manual replacements. Read [robots.txt vs llms.txt](https://www.answer.cloud/pages/ai-crawling-permissions-robots-txt-vs-llms-txt-hosted/).