Get the latest tech news
Meta unleashes new web crawling bots with sneaky ways of avoiding a rule that blocks scraping of online content
Meta's new AI bots, Meta-ExternalAgent and Meta-ExternalFetcher, scrape web data and may bypass robots.txt rules.
A second one, called Meta-ExternalFetcher, is related to the company's AI assistant offerings and collects web links to support specific product functions. These bots first appeared some time in July, according to archived Meta web pages analyzed by Originality.ai, a startup that specializes in spotting AI content. Website owners may wish to block Meta from sucking up their data for AI model training, but they may want the tech giant to index their sites so more human users visit.
Or read this on r/technology