A new restriction is coming for the tech world as a major player, Cloudflare, has announced that its AI agent crawlers will be blocked by default starting September 15th. This change affects a specific slice of the web and follows Google's own announcement just over a month ago regarding similar restrictions on its services.
The decision comes after concerns were raised about the potential misuse of AI agents to scrape pages from websites without permission, raising questions about their use in search algorithms and other online applications. Cloudflare is not alone in implementing this change; most coverage has focused on Google's own efforts towards curbing these practices.
To ensure that your content continues to appear as intended on the web, users can opt-in for AI agent crawlers by adding a meta tag called "robots.txt" to their website. Alternatively, developers can implement the necessary measures to prevent the bots from accessing their sites in the first place.