Get the latest tech news

Websites accuse AI startup Anthropic of bypassing their anti-scraping rules and protocol

iFixit and Freelancer have accused Anthropic, the AI startup behind the Claude large language models, of ignoring their "do not crawl" robots.txt protocol and policy to scrape their websites' data.

Freelancer has accused Anthropic, the AI startup behind the Claude large language models, of ignoring its "do not crawl" robots.txt protocol to scrape its websites' data. After Wired's piece came out, a startup called TollBit that connects AI firms with content publishers reported that it's not just Perplexity that's bypassing robots.txt signals. iFixit's Wiens seems open to the idea of signing a deal for the how-to-repair's website's articles, as well, telling Anthropic in a tweet he's willing to have a conversation about licensing content for commercial use.

Get the Android app