Get the latest tech news

A look at search engines with their own indexes (2021)

A cursory review of all the non-metasearch, indexing search engines I have been able to find.

Feel free to send me suggestions, updates, and corrections; I’d especially appreciate help from those who speak languages besides English and can evaluate a non-English indexing search engine. Semantic Scholar A search engine by the Allen Institute for AI focused on academic PDFs, with a couple hundred million papers indexed. Too many sites block perfectly well-behaved crawlers, only allowing major players like Googlebot, BingBot, and TwitterBot; this cements the current duopoly over English search and is harmful to the health of the Web as a whole.

Get the Android app