AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare announced this change earlier this month and most of the coverage since then has focused on Google. The reason behind the change is to prevent abuse of these crawlers.
Cloudflare explains that these agents are often used in spam detection and anti-scraping measures, which can be problematic when they're used too aggressively or without permission from the websites they're accessing. By blocking AI agent crawlers by default, Cloudflare aims to create a safer online environment for everyone.
To get an AI agent crawler up and running on your website, you'll need to obtain the necessary permissions first. This typically involves obtaining specific licenses or signing up with data brokerage firms that offer permissioned crawling solutions. Depending on the type of crawling and the websites involved, this can be a complex process that may require consultation with technical experts.