Get the latest tech news

AI search engines that don’t pay up can’t index Reddit content

When Reddit said last month that it would block unauthorized data scraping from its site, everyone’s (rightful) first reaction was “AI.” However, now that the change has taken effect, search engines that don’t have deals with the website now appear to be locked out.

The widely used forum also appears to be blocking all search engines other than Google, which reportedly inked a deal earlier this year with Reddit worth $60 million annually. When Reddit said last month that it would update its Robots Exclusion Protocol (robots.txt) to block automated data scraping, it’s now apparent that it wasn’t only meant to thwart AI companies like Perplexity and its controversial “answer engine.” Currently, Google appears to be the only search engine allowed to crawl Reddit and produce results from “the front page of the internet.” Ironically, part of the forum website’s robots.txt file reads, “Reddit believes in an open internet, but not the misuse of public content.” The file for Reddit now essentially says, “Do not scrape.” Apparently, it now considers search engines that don’t buy into exclusive deals to be misusing its content.

Get the Android app