AI agents, those clever bots that fetch pages in real-time for users waiting for answers, have been a norm online for years. However, their newfound importance has come with a twist - they will be blocked by default from September 15 onwards due to a change made by Cloudflare on July 1.
This change is part of a broader effort to improve the security and control of the internet, as more AI-powered websites become increasingly prevalent online. While some have welcomed this move, others have raised concerns about the potential impact on users relying on these bots for essential services. The change affects an unspecified portion of the web, leaving many uncertain about how it will affect their browsing habits.
To get around the blocked crawler, developers can apply for permission to use these AI agents on a case-by-case basis. This may involve submitting an application to Cloudflare's support team or working with other companies that have already secured permissions from the platform. The move is seen as a response to growing concerns about the potential risks associated with widespread adoption of AI-powered websites.