Cloudflare Lets Sites Block AI Training While Keeping Search Visibility

For years, website owners faced an impossible choice: let AI training crawlers access their content, or block them entirely and disappear from search results. The two functions—indexing for search engines and harvesting material to train artificial intelligence systems—often traveled together in the same crawler, bundled as a package deal. Block one, and you blocked both. That binary trap is what Cloudflare set out to solve.

The company has rolled out a new setting called Disallow AI Training that lets site owners refuse access to AI training operations while remaining fully visible to search engines. The mechanism works by distinguishing between crawlers that respect the choice and those that don't. If a crawler honors the instruction to stay out of AI training, it gets through to the site. If it ignores the request, Cloudflare blocks it entirely. The distinction matters because some of the largest technology companies—Apple, Google, and Microsoft—have either already aligned their crawler behavior with this standard or committed to do so in the near term.

The problem Cloudflare identified was real and widespread. Many modern crawlers serve dual purposes: they feed search indexes while simultaneously collecting training data for machine learning models. A website owner who wanted to prevent their work from being used to train AI had no surgical option. They could only slam the door on the entire crawler, which meant their site would vanish from search results. That all-or-nothing dynamic left many site owners trapped between two undesirable outcomes, neither of which served their interests.

Cloudflare's approach sidesteps that trap by creating a middle ground. The company has been in conversation with crawler operators since July about the issue, according to Bryan Becker, who directs product work on security and trust solutions at Cloudflare. The response from the industry has been largely receptive. Nearly every operator Cloudflare spoke with acknowledged that site owners should have meaningful control over how their content gets used and that those choices should be respected and enforced.

The support from major players matters. Apple, Google, and Microsoft collectively control a significant portion of the crawler ecosystem. Their willingness to meet Cloudflare's criteria—or their public commitment to do so—gives the new setting real teeth. Several other companies already operate separate crawlers for search and training, which means Cloudflare can block their training crawlers without touching the search ones. That existing separation makes the technical problem easier to solve for those operators.

Matthew Prince, Cloudflare's co-founder and chief executive, framed the move as a way to preserve the internet's openness while giving creators and businesses genuine agency. The language reflects a broader tension in the current moment: search engines depend on open access to content to function, but that same openness has created a pathway for AI systems to absorb vast amounts of material without explicit permission or compensation. Cloudflare's solution tries to honor both needs—keeping search alive while letting creators opt out of AI training.

The timing connects to a wider industry reckoning with bot activity and automated access. A recent PYMNTS Intelligence report found that nearly nine in ten organizations now struggle with managing bot traffic on their platforms. As AI agents become more sophisticated and more prevalent, the challenge of distinguishing between legitimate automated access and harmful scraping has grown sharper. Weak or outdated controls allow malicious bots to operate while legitimate users get blocked by mistake. Cloudflare's new setting is one piece of that larger puzzle—a way to give site owners more granular control over who accesses their work and for what purpose.

What happens next depends partly on adoption. Site owners need to know the setting exists and understand how to use it. Crawler operators need to continue respecting the choices that site owners make. The industry conversation that began in July will likely continue, with other crawler operators potentially facing pressure to align their behavior with what Apple, Google, and Microsoft are doing. For now, the framework is in place. Whether it becomes the standard or remains a niche tool depends on how broadly it gets implemented and whether site owners actually use it.