Get the latest tech news

Perplexity AI is lying about their user agent

Perplexity AI claims it sends a user agent and respects robots.txt but it absolutely does not

March 30th: Started disallowing PerplexityBot, and others, in my robots.txt June 14th: Added server-side blocking in nginx - any match would result in a 403 forbidden response I should not have provided a summary of the blog post at https://rknight.me/blog/blocking-bots-with-nginx/ since that website's robots.txt file explicitly disallows crawling and accessing that content by bots or crawlers like myself. 👋 Found Perplexity from a list of AI bots and noticed it's not respecting robots.txt or sending the correct user agent so it can be blocked by servers

Get the Android app