Get Permission for AI Agent Crawlers
A crucial step in harnessing the potential of AI technology is getting permission for AI agents to crawl websites, a process that has been taken by Cloudflare from September 15 onwards. This change will block most AI agent crawlers from accessing certain slices of the web. The decision was announced by Cloudflare on July 1.
The new policy affects many online platforms and services, including those reliant on Google for crawling pages in real-time. Since then, numerous articles have focused primarily on Google's response to this change, which is not directly relevant to the question at hand. However, it does highlight the importance of considering the impact of such changes on various stakeholders.
To address the need for permission from AI agents, users can opt-in to allow certain websites or services to crawl their pages. This process typically involves adding specific URLs to a list of whitelisted sites in the settings or configuration page of the website's crawler software. By implementing this step-by-step guide, individuals and organizations can ensure that their online presence is not disrupted by these changes.