Get the latest tech news

Robots.txt is a suicide note (2011)

OBOTS.TXT IS A SUICIDE NOTE ROBOTS.TXT is a stupid, silly idea in the modern era. Archive Team entirely ignores it and with precisely one exception, everyone else should too.

For the unfamiliar, ROBOTS.TXT is a machine-readable textfile that sits on webservers that gives instructions as to what items, directories or sections of a web site should not be "crawled", that is, viewed by search engines or downloaded via programs, or otherwise accessed by automatic means. The purpose and meaning behind the creation of ROBOTS.TXT file dates back to the early 1990s, when the then-new World Wide Web was quickly defining itself as the killer application that would change forever how users would interact with the growing internet. Where previous information networks utilizing internet connections such as GOPHER and WAIS were text-based and relatively low-bandwidth, the combination of text, graphics and even sounds on webpages meant that resources were stretched to the limit.

Get the Android app