Get the latest tech news
Wikimedia Drowning in AI Bot Traffic as Crawlers Consume 65% of Resources
Web crawlers collecting training data for AI models are overwhelming Wikipedia's infrastructure, with bot traffic growing exponentially since early 2024, according to the Wikimedia Foundation. According to data released April 1, bandwidth for multimedia content has surged 50% since January, primaril...
Web crawlers collecting training data for AI models are overwhelming Wikipedia's infrastructure, with bot traffic growing exponentially since early 2024, according to the Wikimedia Foundation. According to data released April 1, bandwidth for multimedia content has surged 50% since January, primarily from automated programs scraping Wikimedia Commons' 144 million openly licensed media files.This unprecedented traffic is causing operational challenges for the non-profit. The foundation's Site Reliability team now routinely blocks overwhelming crawler traffic to prevent service disruptions.
Or read this on Slashdot
