Get the latest tech news

FOSS infrastructure is under attack by AI companies

LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.

AI scrapers are getting more and more aggressive, and - since FOSS software relies on public collaboration, whereas private companies don't have that requirement - this is putting some extra burden on Open Source communities. According to Ben, part of the KDE sysadmin team, all of the IPs that were performing this DDoS were claiming to be MS Edge, and were due to Chinese AI companies; he mentions that Western LLM operators, such as OpenAI and Anthropic, were at least setting a proper UA - again, more on this later. I do wonder how much of this is scraping for training data, and how much instead is the "search" function that most LLMs provide; nonetheless, according to Schubert, "normal" crawlers such as Google's and Bing's only add up to a fraction of a single percentage point, which hints at the fact that other companies are indeed abusing their web powers.

Get the Android app