Get the latest tech news
AI scrapers running out of space as restrictions close the net
AI scrapers increasingly face hostile online environments as data sources dry up a new study from has found.
14,000 web domains were assessed with the discovery of an “emerging crisis in consent” as online publishers have reacted to the presence of crawlers and the harvest of data. The study found between 20 and 33 percent of the top web domains have introduced extensive restrictions on scrapers, compared to a much lesser figure at the start of last year. It was noted that many websites had changed their terms of service to completely prohibit crawling and lifting content for use in generative AI, but not to the extent of the restrictions on robot.txt.
Or read this on ReadWrite
