Unlocking AI is a complex process that relies heavily on artificial intelligence agents, or bots, which fetch pages in real time on behalf of users waiting for an answer. These agents are widely used in various applications such as search engines, social media platforms, and online services. However, they have been subject to restrictions since July 1 when Cloudflare announced a change to block AI agent crawlers.
The restriction is aimed at reducing the load on the cloud infrastructure of websites hosting these bots. According to Cloudflare, most coverage of this change has focused on Google, with some media outlets covering other tech companies like Amazon and Microsoft. However, it appears that only those tech companies have been affected by the change so far.
To get AI agent crawlers to appear in a website's search results or other applications without permission, users can try adding a "noindex" meta tag to their website. This tag instructs web crawlers not to index the content of the page. Additionally, some online services such as Google provide tools that allow users to request the removal of these agents from their websites.