Get the latest tech news
Anthropic’s crawler is ignoring websites’ anti-AI scraping policies
iFixit’s CEO says ClaudeBot tied up its DevOps resources.
The ClaudeBot web crawler that Anthropic uses to scrape training data for AI models like Claude has hammered iFixit’s website almost a million times in a 24-hour period, seemingly violating the repair company’s Terms of Use in the process. iFixit doesn’t seem to be alone, with Read the Docs co-founder Eric Holscher and Freelancer.com CEO Matt Barrie saying in Wiens’ thread that their site had also been aggressively scraped by Anthropic’s crawler. Disallowing crawlers via robots.txt files is also the opt-out method of choice for many other AI companies like OpenAI, but it doesn’t provide website owners with any flexibility to denote what scraping is and isn’t permitted.
Or read this on The Verge