Google crawls 3X more of the web than OpenAI, & how that matters

Google's dominant position in crawling the web may allow it to remain head of its competitors even in the AI race. This was revealed by recent data shared by Cloudflare CEO Matthew Prince.

According to that, Google crawls 3.2X more web pages than OpenAI, its closest competitor.

Speaking at a podcast, Prince said that this gap further widens in the case of other competitors. The search giant accesses 4.6 times more content than Microsoft and nearly five times more than Anthropic or Meta. Web crawling is the process by which software bots scan the worldwide web for data.


"What I worry about is, because Google has this unique access to the web that nobody else has, the game might just go to them. Because at the end of the day, whoever has the most data wins in the era of AI,” said Prince in a podcast that premiered on Saturday.

This is not the first time that Prince has pointed to Google’s hegemony in web crawling. In a November 2025 post and at various public appearances, Prince has voiced his concerns around this. He has argued that Google’s advantage stems from tightly linking its search crawler with its AI data collection.

For publishers, this creates a stark choice which entails either blocking Google’s crawler and risk disappearing from search, or allow their content to be used for AI training.

In contrast, many other AI companies identify their crawlers separately, making them easier for site owners to block.

Since July 1, 2025, Cloudflare claims to have blocked 416 billion AI-related requests on behalf of its customers, largely affecting non-Google players who follow crawler disclosure norms.

Without intervention, Prince believes Google’s historic dominance in search could translate into long-term dominance in AI, leaving competitors unable to catch up unless the data playing field is levelled.

“Either (we) bring Google back to where everyone else is, don't let them leverage search in order to get a unique advantage in AI, or make sure that everybody has access to that same data,” Prince added.