AI agent crawlers, those bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. This change was announced by Cloudflare on July 1 and has been the subject of widespread coverage since then.
Most of the coverage focused on Google, but it's worth noting that the issue affects all websites that utilize these bots. The exact reason behind this decision is unclear, but it's likely related to concerns about spam and the spreading of misinformation.
To get AI agent crawlers blocked from your website, you can add a simple script to the beginning of your HTML. This will cause the crawler to require permission before accessing any data on your site. You can find more information on how to do this by visiting Cloudflare's documentation on the issue.