Get the latest tech news
Cloudflare is arming content creators with free weapons in the battle against AI bot crawlers
Cloudflare is providing tools that give website owners more control over who can access their data, as well as the ability to analyze how their content is used by AI models.
In just one example, repair database iFixIt complained in July that a web crawler bot for Anthropic’s AI chatbot Claude hit its website nearly a million times in a single day. Cloudflare maintains that its analytics can also help those signing licensing agreements with model providers understand the metrics used in negotiations, such as the rate for crawling certain sections or the entire page. While large publishers may strike direct deals, the AI model providers will struggle to access high-quality content from smaller websites.
Or read this on r/technology
