Microsoft Clarity has added a metric that compares how much AI crawlers scrape from a site with how many visitors they send back. The AI Scrape-to-Referral Ratio, now in Clarity’s Bot Analytics dashboard, pairs bot-crawl data with referral analytics in a single card. In Microsoft’s own example, the average ran about 6,000 scrapes for every referral.

The card comes with a ranked breakdown of which AI operators send traffic and which mostly take, and it links into filtered session records, so a marketer can see what an AI-referred visitor does after landing, Clarity limits the math to correctly mapped domains, so a coverage mismatch doesn’t distort the number.

Why the ratio matters

The metric speaks to a growing debate over what AI companies owe the sites they crawl. In a July 2025 report, Cloudflare, which now blocks AI crawlers by default and runs a pay-per-crawl marketplace, found that Anthropic's crawler made nearly 71,000 requests for every one visitor it referred over a single week in June. Cloudflare cautions the figure can overstate the full picture, as traffic from AI apps often carries no referrer, but the point remains: the models take more and send back the same or less.

"AI crawlers have been scraping content without limits," said Cloudflare’s CEO Matthew Prince, and set Cloudflare's goal as putting "the power back in the hands of creators."

The ratio reflects a shift in what marketers measure. Most tools count how often a brand appears in AI answers. The AI Scrape-to-Referral Ratio tracks what those appearances send back in traffic. Semrush projects that AI search visitors will overtake traditional search by 2028.

What it can and can’t tell you

Clarity's ratio is a close-up of one site. Cloudflare measures the same gap across its whole network; Clarity measures it on your domain and ties it to recordings that show what those visitors do.

But the number has two blind spots:

1. A high scrape count with few referrals doesn’t itself signal a failing AI visibility because an AI answer can build awareness without ever sending a click
2. As Cloudflare reported, the referral side is likely undercounted: AI platforms often pass no referrer, so their visits are logged as direct traffic in analytics

How to act on it with Semrush

Clarity shows which operators take and which give back on your own domain. It doesn't show where you stand in the AI answers themselves, or how you compare with competitors.

Semrush's AI Visibility tools track brand mentions, citations, and share of voice across ChatGPT, Google AI Overviews, Google AI Mode, and Gemini at the prompt level, so you can see which prompts surface you versus a competitor.

Where Clarity flags a page that gets scraped but returns little, Content Optimizer recommends changes to its structure and completeness for both traditional and AI search.

Site Audit surfaces the crawlability, indexing, and structured-data issues that can keep AI platforms from parsing your content.

For enterprise teams tracking AI visibility at scale, Semrush Enterprise can connect AI search visibility with traffic and conversion data to measure revenue impact. It also incorporates traffic logs, AI visibility data, and content optimization recommendations to help teams understand both how AI systems access their content and how that visibility contributes to business outcomes.

Beat your competition in LLMs

with the Semrush AI Visibility Toolkit

Sign up now

Share

Most popular pages