AI agent crawlers have long been a staple in online search, allowing users to fetch pages in real-time as they type or wait for an answer. However, the industry has taken notice of these bots, and Cloudflare is now introducing changes that will impact their use.
As of September 15, AI agent crawlers will be blocked by default on a slice of the web. This means that any website relying on these bots will need to obtain permission before they can continue operating. The decision was announced in July, and most coverage has focused on Google's efforts to regulate the industry. However, this change applies broadly, and affected websites must now seek approval from their hosting providers or content owners.
While some have welcomed the regulation, others see it as an overreach by Cloudflare. "This is a classic example of an information asymmetry problem," said one industry insider. "Cloudflare has been making decisions without considering the broader implications for users and publishers alike." With the change now in effect, affected websites must navigate a complex web of permissions and policies to continue operating, raising questions about what this means for online search and commerce.