Get the latest tech news
Nearly 90% of our AI crawler traffic is from ByteDance
TikTok’s web scraper, Bytespider, is aggressively sucking up content to fuel generative AI models. Here's what we've learned from our bot management analytics.
Content-scraping bots existed long before LLMs started crawling the web for generative AI applications, and they have usually been considered undesirable visitors on content-heavy websites. Other businesses running content-heavy public websites will likely find themselves having to make the same decision: to protect the value of their content, or to allow the dissemination of information about their brand and products via these new channels. Therefore, any technical solution for managing AI crawlers and scrapers must be capable of accurately identifying such bots, even when they are designed to be hard to distinguish from humans.
Or read this on Hacker News