Bots passed humans on the open web in June 2026, 18 months ahead of schedule — and the crawler taking the most from your site still gave back one visit for every 1,782 pages it took in August
For the first time in the internet's history, more requests hitting the open web now come from automated traffic than from people — and a meaningful share of that automated traffic is AI crawlers reading your site to answer someone else's question, not to send you a customer. The ratio is improving, but it isn't close to even, and almost nobody has looked at what their own server logs — or their robots.txt file — actually say about it.
Somewhere on your website right now, a crawler is reading your pricing page, your service pages, your case studies — the pages you wrote so a customer would find them and call. Most of the time, no customer is on the other end of that request. It's a model being trained, or an AI assistant answering someone's question using your content, without sending that person to you. Your server pays for the request either way — the bandwidth, the compute, the log line — whether or not anyone ever arrives.
This used to be a rounding error. It isn't one anymore. On June 3, 2026, Cloudflare — which sits in front of roughly a fifth of the web — reported that automated requests had overtaken human ones for the first time ever: 57.5% bot, 42.5% human. Cloudflare's own CEO had predicted that crossover for late 2027. It arrived a year and a half early.
This piece is about what's actually inside that bot majority — which crawlers take the most and give back the least, how fast that's changing month to month, and what a business is actually deciding when its robots.txt file was never touched after launch. It isn't an argument to block everything or allow everything. It's the math you need before you pick either.
Start with the number that made this a story instead of a trend line: bots passing humans on the open web. Cloudflare Radar's June 2026 measurement — taken across the roughly one-fifth of all websites that sit behind Cloudflare — put automated HTML requests at 57.5% against 42.5% from actual people [1]. Cloudflare's own leadership had modeled this crossover for the end of 2027; the growth of agentic AI traffic, autonomous programs browsing the web on behalf of an assistant, pulled it forward by about a year and a half [1]. Not all of that bot share is AI — search indexers, uptime monitors, and security scanners were always part of the mix — but the AI portion is the part that grew fast enough to move the date.
The more useful number for a business isn't the bot share of traffic — it's what any single crawler gives back for what it takes. Cloudflare Radar tracks this as a crawl-to-refer ratio: pages a platform's bot reads, divided by visitors that platform's consumer product actually sends to the site. In June 2026, Anthropic's ratio stood at roughly 4,580:1 — 4,580 pages read for every one visitor referred. OpenAI sat at 848:1, Perplexity at 186:1, and Google, whose core product depends on sending people to websites, at roughly 5:1 [2].
That ratio moves fast, and not always in the direction you'd expect. By the trailing 28 days to August 1, 2026, Anthropic's ratio had fallen to about 1,782:1 — a roughly 61% improvement in two months — and OpenAI's to 233:1, tracking with both platforms sending more traffic back through their consumer products, not with either one crawling less [3]. Perplexity moved the opposite way, from 186:1 to 303:1, a reminder that no platform's number is fixed — it reflects a product decision on their end that can reverse without warning. Google held close to break-even across the same window, near 4.79:1, the one major platform whose incentives were built around referral traffic from the start [3].
The practical question this raises for any business is what its own robots.txt file is actually doing, and whether that was ever a choice. Whole-web data on the file puts the picture in context: only about 7.9% of all sites block OpenAI's crawler outright, but that jumps to roughly 50.5% among publishers — the businesses whose entire product is the content itself. Blocking any AI crawler at all sits near 10.3% for the general web against 56.4% for publishers, and roughly 30% of the top sites now run what amounts to a middle path: block the bots that only train a model, allow the ones that can still cite you and send someone back [4]. ClaudeBot's specific share of block rules kept climbing through the same period, from 9.6% in January 2026 to 10.1% by March [4] — evidence that this is an active, ongoing decision for the businesses paying attention to it, not a one-time setting.
None of this is a reason to panic-block every AI crawler your logs show you. A crawler that only trains a model on your product pages and never cites you back is a pure cost with no return path — blocking it loses nothing you had. A crawler that reads your pages in order to answer a prospective customer's question and name your business while doing it is the newer version of the exact traffic your SEO work has always chased — blocking it on principle gives away a citation for nothing. The businesses getting this right aren't running one blanket rule; they're reading their own logs to see which bots show up, checking each one's referral behavior against reports like Cloudflare's, and setting robots.txt and llms.txt rules by purpose instead of by name recognition.
The instinct is understandable: nobody signed up to have their product pages read thousands of times for nothing. But a blanket block also closes the door on the crawlers that do cite you — Google's 5:1 ratio exists because its business model rewards sending people to sites, and an AI answer that names your business by name is a referral channel, not a threat. Block indiscriminately and you lose the upside along with the cost.
The opposite instinct — stay open to every crawler on the theory that more AI visibility can only help — ignores that roughly half the platforms crawling you send back almost nothing for it. Anthropic's ratio improved 61% in two months and is still over 1,700:1. Being read is not the same as being found, and a business that never checks the difference is paying a real hosting bill for an outcome it never measured.
There's a real version of this argument: if bots are now 57.5% of your HTML traffic, that's compute and bandwidth you're paying for regardless of the strategic angle. True — but the fix for a cost problem and the fix for a visibility problem are the same file. A robots.txt written by purpose, not by blanket allow or blanket deny, addresses both at once.
Read your own logs before you touch the robots.txt file. Separate crawlers by what they actually do with what they take — a training-only bot that never refers anyone gets blocked; a bot behind an answer engine that names and links your business gets allowed, and gets a clean, structured version of your content (schema markup, an llms.txt file, plain factual pages) so it has something worth citing. Check the ratio again in a quarter — Perplexity's own number moved backward in two months, and yours can too.
- 01Do you know what share of your site's traffic is a crawler rather than an actual visitor — or would you only find out from a hosting bill that jumped for no obvious reason?
- 02If a bot read your product pages 1,782 times last month and sent back one visitor, is that a trade you'd have made on purpose, or one that just happened because nobody set the file?
- 03Is your robots.txt file blocking or allowing AI crawlers by a decision someone made, or is it still whatever a template shipped with when the site went live?
- 04If you block every AI crawler at the door while your content stays unstructured and hard to cite for the ones you'd actually want reading it, are you protecting anything or just closing every door equally?
- 05Given that Perplexity's ratio moved the wrong way in two months while Anthropic's improved, is this a setting you check once, or one somebody actually revisits?
- [1]Cloudflare Radar, via Matthew Prince (CEO), X post, June 3, 2026: automated HTML requests reached 57.5% of traffic against 42.5% human, across the roughly one-fifth of the web that sits behind Cloudflare — a crossover Cloudflare's own leadership had forecast for late 2027, arriving about 18 months early on the growth of agentic AI browsing traffic.
- [2]Cloudflare Radar — "The crawl before the fall… of referrals: understanding AI's impact on content providers" (June 2026 measurement): crawl-to-refer ratios of 4,580:1 for Anthropic, 848:1 for OpenAI, 186:1 for Perplexity, and roughly 5:1 for Google.
- [3]SEOmator — "GEO Data Report 2026: Which AI Crawlers & LLM Bots Take the Most and Give the Least?" — month-over-month Cloudflare Radar tracking: Anthropic 4,580:1 (June) to 1,782:1 (trailing 28 days to August 1); OpenAI 848:1 to 233:1; Perplexity 186:1 to 303:1; Google roughly 5:1 to 4.79:1; Mistral peaking near 3,389:1 in July.
- [4]Presenc.ai — "State of Robots.txt for AI 2026: Adoption Tracker": whole-web GPTBot block rate 7.9% versus 50.5% among publishers; combined "blocks any AI crawler" rate 10.3% whole-web versus 56.4% for publishers; roughly 30% of top sites now block training-only bots while allowing search/answer bots; ClaudeBot's share of robots.txt DISALLOW rules rose from 9.6% (January 2026) to 10.1% (March 2026).
We set your crawl-access policy on purpose — open to the bots that send customers back, closed to the ones that only take.
Felukaa builds a differentiated robots.txt, structured schema markup, and an llms.txt file into every site and research page we ship — so the crawlers worth allowing have something worth citing, and the ones that only train a model aren't given free run of your server. If you don't know what's actually reading your site right now, we'll show you from your own logs before we touch a line of code.
Book a free 15-min consultation