Get Permission First! Unlock Access to the Powerful New AI Agent Crawlers
Cloudflare, a leading technology company, has announced that its AI agent crawlers will be blocked by default on a slice of the web starting from September 15. This change was made as part of an update to prevent abuse and ensure the responsible use of these powerful bots.
The AI agents are designed to fetch pages in real-time for users waiting for answers or information, but their primary function has been exploited by malicious actors. To unlock access to this technology, users need to obtain permission from Cloudflare first. This will involve registering a domain with Cloudflare and obtaining an API key that allows the crawler to operate.
As the announcement on July 1 highlighted, Google was one of the most notable companies affected by the change. Users who want to use AI agent crawlers can now register their domains through Cloudflare's platform, which enables them to unlock access to these powerful bots while ensuring they are used responsibly and within legal bounds. By obtaining permission first, users can ensure that their usage complies with existing regulations and protects both themselves and others from potential misuse of the technology.