Bots Passed Humans. The Web's Business Model Is Next.

A Line Got Crossed, and It Is a Business-Model Event

For the whole history of the web, humans were the traffic. You built a page, people came to read it. That is over.

In its new report Content Independence Day, one year on, Cloudflare, which sits in front of more than 20% of the web and can actually count, reports that agent traffic has crossed a historic threshold: more than half of all traffic on the Internet is now non-human. Cloudflare’s own line is that the shift happened faster than they anticipated.

Treat that as a curiosity and you miss the point: this is the moment the web’s business model stopped matching the web’s traffic. And when the economics of a platform break, value moves. For anyone allocating capital, the interesting question is “where does the value go, and which of my companies are standing in the wrong place?”

The Exchange That Funded the Web Is Breaking

The open web ran on one deal, whether or not anyone wrote it down. You publish content. Search engines crawl it. In return, they send you humans. The humans see ads, buy products, become customers. Crawl in, traffic back. That loop paid for journalism, documentation, reviews, forums. Most of what is worth reading.

AI agents broke the loop. They crawl the content and send almost nothing back.

The report is blunt about the shift. As of June 2026, 52% of crawler requests are for AI training (up from 22% a year earlier), while pure search crawling, the kind that used to return visitors, is a small and shrinking slice. Meanwhile human attention is draining from the open web: for every hour people spend looking for information, only 15 minutes is now spent on the open web. The rest resolves inside an AI answer. The source gets a training pass and no click.

The old web sent you visitors in exchange for your content. The agentic web takes the content and keeps the visitor.

This is not a media problem. It started with news and publishing, but Cloudflare reports that some of the most heavily crawled categories (retail, software, IT, finance) have already seen human traffic fall by as much as 40% in under a year. Publishers now plan for “Google Zero”: a world where essentially no traffic arrives from search referrals. The distribution assumption underneath a great many business plans just changed, quietly, without the plan being updated.

Watch the Gatekeeper

There is a concentration risk in here that an investor should not miss. Google still accounts for roughly 88% of referral traffic: it is the gateway. But Google is increasingly answering queries inside its own AI surfaces instead of sending the click onward. And unlike other AI companies, which let you separate their search crawler from their training crawler, Google runs a single mixed-use bot: you cannot stay in Google’s search index without also feeding Google’s AI. Cloudflare estimates that gives Google access to about twice the content of other AI companies.

So the channel a huge share of the web still depends on for discovery is the same channel quietly absorbing the visit. That is not a diversified risk. For any company whose growth rests on Google organic, it is a single point of failure that is actively turning against the traffic it provides.

The Internet Is Being Re-Plumbed for Machines

Here is the part that turns a threat into a map. A market is forming around the new reality, and Cloudflare’s year-one data shows its shape: transparency created scarcity, scarcity created leverage, leverage is producing licensing. More than 50 publisher-AI licensing agreements have been signed since 2023, and the debate has moved from whether content should be paid for to how.

When agents are the customer, value stops accruing to whoever wins human attention and starts accruing to whoever the agents cannot route around:

• Proprietary data worth licensing. If a model must have your content and cannot synthesize it elsewhere, you have pricing power in the new market. If your content is commodity, you were only ever paid in traffic, and the traffic is leaving.
• Being the answer layer, not a source for it. The product an agent calls (an API, a live inventory, a transaction endpoint) captures value. The page an agent reads and discards does not.
• Machine-readable commerce. Agents that book, buy and transact need checkout, auth and identity built for them, not for a human squinting at a form. Cloudflare is explicit that the next phase needs new infrastructure for permissions, licensing and transactions at scale, and that infrastructure is thin today.
• Brand and direct relationships. The one demand an agent cannot disintermediate is the customer who came for you by name. Owned audience stops being a nice-to-have and becomes the moat.

The Question to Ask in Diligence

For an investor, this collapses to one uncomfortable question about any company touching the web: is it built to be consumed by a human with a browser, or by an agent with an API?

If the answer is “human with a browser,” and the growth model leans on organic search or referral traffic, you are underwriting a distribution channel that is actively eroding, and the erosion does not show up in this quarter’s numbers, which is exactly what makes it dangerous. The metrics look fine right up until the referral base gives way.

The sharper diligence is concrete. What share of acquisition depends on Google organic, and is that traffic already softening as AI answers absorb the query? Is the company’s content a licensable asset or a commodity being scraped for free? Can its product be called by an agent, or only used by a person? Is anyone on the team even measuring bot versus human traffic, or is a growing share of their “engagement” already machines they cannot see?

Now the honest caveat, because this gets oversold in both directions. Cloudflare says it plainly: licensing today is bespoke and “unlikely to fully replace lost referral, advertising, and affiliate revenue,” and content valuation is still unresolved. Humans still buy. Brand still converts. The new rails are unsettled, and the winners among licensing, protocols and agent-commerce are not decided. The risk is real but the timing is not a switch; it is a slope. Betting the collapse happens next quarter is as wrong as pretending it will not happen. The job is to underwrite the direction and price the timing honestly.

Why This Is Our Work

When we assess a portfolio company’s engineering organization, this is no longer an abstract macro trend: it is a line item in technical due diligence. Is the product architected for the agentic web or the human web? Can it be called, licensed, transacted against by a machine, or does its entire distribution rest on a referral loop that is thinning? Those are engineering questions with valuation consequences, and most diligence never asks them.

We do both halves. We assess the dev org against where the web is actually going, and then, same team, same engagement, we build the capability to move it there. Not a report that names the risk and leaves. A hundred days of working alongside the team to make the product something an agent can reach, and to leave that capability behind when we go.

The web is being rebuilt for machines. The portfolios that read the shift early, and re-tool for it deliberately, will own the next decade of it. The ones that keep optimizing for a human who is no longer clicking will not.