AI tools can retrieve HTML and plain-text resources, including Markdown, when their tools support the destination and format. Behavior varies by product and task.
Separate content from discovery files
Robots.txt expresses crawler preferences. A sitemap lists URLs. An llms.txt file is a proposed content guide. A text mirror contains the page’s information in a simpler representation. Requests to these different resources should not be treated as equivalent.
Avoid a universal preference claim
Private logs from one website do not prove that all search bots prefer .txt, all user fetchers prefer HTML, or .md performs better everywhere. A comparison needs a defined sample, date range, request categories, and outcome measure. A fetch is not a citation.
Publish useful HTML first and keep any text versions consistent with it. answer.cloud generates text mirrors and discovery resources through the publishing workflow. Check your own requests and controlled retrieval tests to see which resources are used.
Read the llms.txt guide and metrics guide.