Get the latest tech news
The Race to Block OpenAI’s Scraping Bots Is Slowing Down
OpenAI’s spree of licensing agreements is paying off already—at least in terms of getting publishers to lower their guard.
When companies enter into partnerships and give permission for their data to be used, they’re no longer incentivized to barricade it, so it would follow that they would update their robots.txt files to permit crawling; make enough deals and the overall percentage of sites blocking crawlers will almost certainly go down. Whatever happens, this is a revealing moment: While publishers had initially responded to the rise of AI scraping bots with the shared impulse to block them, OpenAI’s active pursuit of partnerships has cooled that industry-wide drive. Kate Knibbs is a senior writer at WIRED, covering the human side of the generative AI boom and how new tech shapes the arts, entertainment, and media industries.
Or read this on Wired
