Skip to content

AI Crawling and Referral Traffic Are Different Signals

Read historical crawl-to-refer charts as request-to-referral ratios, not citation counts or guarantees for your website.

Published

The historical charts below compare crawl requests with detectable referrals. They do not count citations in AI answers. See Cloudflare’s 2025 crawl-to-refer methodology.

Interpret the ratio correctly

The provider-level metric compares HTML crawl requests identified through user-agent headers with HTML referrals identified through referrer headers. A provider can have several agents. It is incorrect to assign the whole ratio to one bot or say a crawler “references your site” a fixed number of times before sending a visitor.

The values describe a historical network sample, not a promise for an individual website. Missing referrers and classification limits affect what can be observed.

Use your own request, referral, and conversion data alongside controlled answer tests. A high crawl count does not establish frequent citations or business value. Read the metrics guide.

Article illustration

Article illustration

Built for the answer era

Build a website people can use & AI can understand.

Build and deploy with your coding agent, or add an AI companion to your existing website.