AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, are set to be blocked by default on a slice of the web from September 15 onwards. Cloudflare, the company responsible for blocking these agents, announced the change earlier this month on July 1.
Cloudflare's decision follows a broader shift towards better user experience and less spamming of search engines with auto-generated content. Most online coverage has focused on Google's efforts to combat fake news, but fewer have looked into the impact of these crawlers on websites and users alike. The main exception is a recent announcement from Cloudflare stating that AI agent crawlers are now in need of permission.
To get an agent crawler approved for your website or application, you can submit a request through Cloudflare's support portal. In order to receive approval, the crawl requests must be labeled with specific criteria and follow certain guidelines. Once a request is granted, the crawler will begin fetching pages on behalf of users who have requested it, allowing them to interact with websites in real-time without any additional effort from their browsers.