AI crawler activity on business websites
Who actually reads a business website on behalf of AI: crawlers collecting training data, indexers building AI search, and agents fetching a page because someone just asked an assistant a question. Measured in the server logs of 10 sites for from 25 September to 22 October.
In short
Across the sites, 66% of AI agent requests came from training crawlers, 29% from AI search indexers, and 5% from live fetches made for a user. 12% of AI agent requests received an error response. A further 7.5% of requests that named an AI agent were scanners in disguise and are excluded. All figures are shares, and each site counts equally.
Three reasons an AI agent visits
| Purpose | What happens | Share |
|---|
| Model training | A crawler collects pages for a future model. Nothing links back. | 66% |
| AI search indexing | An index is built that an assistant later answers from, often with a link to the source. | 29% |
| Live answer | A person asked an assistant something, and it fetched the page right then to answer. | 5% |
The third line is the one closest to an actual visitor: each request like this is a question being answered using your page as a source. It is also the smallest segment, so it is worth watching week by week rather than as a single number.
Which agents
| Agent | Operator | Purpose | Share |
|---|
| meta-externalagent | Meta | Model training | 36% |
| ClaudeBot | Anthropic | Model training | 15% |
| Applebot | Apple | AI search indexing | 11% |
| OAI-SearchBot | OpenAI | AI search indexing | 8.4% |
| PerplexityBot | Perplexity | AI search indexing | 6.5% |
| Amazonbot | Amazon | Model training | 5.4% |
| GPTBot | OpenAI | Model training | 4.8% |
| Bytespider | ByteDance | Model training | 3.1% |
| ChatGPT-User | OpenAI | Live answer | 2.6% |
| CCBot | Common Crawl | Model training | 1.3% |
| Claude-SearchBot | Anthropic | AI search indexing | 1.2% |
| YouBot | You.com | AI search indexing | 1.1% |
| Claude-User | Anthropic | Live answer | 0.9% |
| DuckAssistBot | DuckDuckGo | AI search indexing | 0.8% |
| Perplexity-User | Perplexity | Live answer | 0.8% |
| meta-externalfetcher | Meta | Live answer | 0.4% |
| MistralAI-User | Mistral | Live answer | 0.3% |
Only agents that identify themselves in the User-Agent header can be counted. Google-Extended and Applebot-Extended are opt-out tokens for robots.txt, with no crawler of their own, so they never appear in a log and are not listed. Classic search engines such as Googlebot and Bingbot are out of scope.
Week by week
| Week starting | Sites | Training | AI search | Live answer |
|---|
| 18.09 | 9 | 60% | 31% | 8.9% |
| 25.09 | 9 | 62% | 29% | 8.3% |
| 02.10 | 9 | 78% | 19% | 3.2% |
| 09.10 | 10 | 62% | 34% | 3.7% |
| 16.10 | 10 | 55% | 39% | 5.3% |
By site
The mix varies widely from site to site, which is why the overall figures give every site equal weight instead of adding up requests: otherwise, one large catalog would dominate the picture.
| Site | Training | AI search | Live answer | Most active agent |
|---|
| slatech.co.il | 79% | 18% | 3.2% | meta-externalagent |
| slavin.ai | 74% | 19% | 7% | meta-externalagent |
| slavin.org.il | 69% | 23% | 8.6% | meta-externalagent |
| slavin.pro | 76% | 17% | 7.7% | meta-externalagent |
| mediora.ai | 48% | 50% | 1.5% | Applebot |
| xfind.me | 77% | 23% | 0% | ClaudeBot |
| darim.ai | 45% | 50% | 5.6% | meta-externalagent |
| yahad.ai | 52% | 46% | 2% | Applebot |
| slatech.ai | 82% | 13% | 5% | meta-externalagent |
| slatech.dev | 58% | 33% | 8.9% | ClaudeBot |
Errors and impostors
12% of AI agent requests received a 4xx or 5xx response, most often because the page no longer exists. For a live answer, that means the assistant answered without your page.
A User-Agent is just a header, and anyone can send one. 7.5% of requests that named an AI agent asked for paths no assistant needs, such as /.env, /.git, or PHP files on sites that have none. These are vulnerability scanners borrowing a trusted name, and they are excluded from every other figure here. Spoofing that asks for ordinary pages cannot be distinguished in these logs.
How it is measured
- Source: IIS access logs from the web servers, read once a day. Days are in UTC.
- A request counts when its User-Agent names a known AI agent. The list of agents and their purpose is included in the downloadable data.
- Each figure is first calculated within each site and then averaged, so every site carries the same weight. No absolute numbers are published.
- Files served by the CDN from cache, such as llms.txt, never reach the server and are invisible here.
- The window covers the last 28 complete days and moves forward every day.
Download
The same figures are available as JSON and CSV, under CC-BY-4.0. Please cite as Slavin AI, AI crawler activity on business websites, with the date of the window.
Frequently asked questions
What share of AI crawler traffic is used for training?
In the current window, 66% of AI agent requests across the measured sites came from training crawlers. The figure is updated daily.
How often do AI assistants fetch a page live for a user?
Live fetches made for a user question accounted for 5% of AI agent requests. They come from agents such as ChatGPT-User, Claude-User, and Perplexity-User.
Can these numbers be faked?
Partly. Anyone can send an AI agent User-Agent. Requests probing for secrets or PHP files are excluded, but impersonation that asks for ordinary pages cannot be detected from logs alone.
Use it on your own site
The same question can be asked of any server log: which agents show up, why, and how many of their requests fail. If a live answer keeps hitting a 404, that page is worth fixing first.
Talk to me
Related: Open data, AI governance, AI glossary