Get the latest tech news
Cloudflare rolls out feature for blocking AI companies' web scrapers
Cloudflare rolls out feature for blocking AI companies’ web scrapers - SiliconANGLE
“Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent,” Cloudflare engineers wrote in a blog post today. One of the crawlers that Cloudflare managed to detect is a bot that collects content for Perplexity AI Inc., a well-funded search engine startup. “When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint,” Cloudflare’s engineers detailed.
Or read this on Hacker News