AI agent crawlers have been a ubiquitous presence online, fetching pages and providing users with real-time answers to their queries. However, they are not as harmless as they seem. As of September 15, the web is expected to be affected by a change in how these bots operate. Cloudflare, a popular internet service provider, has announced that it will block AI agent crawlers on a slice of the web starting from July 1.
The news comes after most coverage of Google's handling of this issue had focused on their efforts to improve crawl limits for the crawler. However, the more useful part of this story is that AI agent crawlers now need permission to operate. This means that website owners will have a responsibility to obtain the necessary consent from users in order to allow these bots to access their content.
To get your AI agent crawler appearing on the web once again, you'll need to implement a solution that allows users to opt-out of having the bot crawl their pages. You can do this by adding an API key or other tracking code to your website's HTML. This will give you permission for the crawlers to access and provide real-time answers to user queries.